Oct 30 01:16:30.329602 ip-10-0-134-254 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:16:30.738877 ip-10-0-134-254 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:16:30.738877 ip-10-0-134-254 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:16:30.738877 ip-10-0-134-254 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:16:30.738877 ip-10-0-134-254 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:16:30.743448 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743358 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:30.743448 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743443 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:30.743448 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743449 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:30.743448 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743453 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:30.743448 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743457 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743460 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743465 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743468 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743474 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743478 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743481 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743485 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743488 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743492 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743495 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743499 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743504 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743525 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743529 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743534 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743537 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743541 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743545 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:30.743753 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743549 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743553 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743557 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743561 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743564 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743568 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743572 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743575 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743579 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743583 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743587 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743594 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743600 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743606 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743610 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743625 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743631 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743636 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743640 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:30.744546 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743644 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743648 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743653 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743657 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743661 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743666 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743670 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743675 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743688 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743696 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743700 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743704 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743708 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743712 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743716 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743721 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743725 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743729 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743733 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743737 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:30.745355 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743741 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743745 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743750 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743753 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743758 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743762 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743766 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743772 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743786 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743791 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743796 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743800 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743804 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743808 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743812 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743816 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743821 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743825 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743829 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:30.746204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743834 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743838 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743850 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743854 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743861 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743865 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743869 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743874 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743878 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743882 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743886 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743891 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743895 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743901 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743908 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743914 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743919 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743923 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743927 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:30.747074 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743932 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743937 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743941 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743946 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743953 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.743957 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745350 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745361 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745367 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745371 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745375 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745379 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745383 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745387 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745391 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745395 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745400 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745405 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745409 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745414 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:30.747644 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745418 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745422 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745426 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745431 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745435 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745439 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745444 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745448 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745453 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745457 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745462 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745466 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745470 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745475 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745479 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745483 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745488 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745492 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745496 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745500 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:30.748391 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745504 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745511 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745516 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745522 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745527 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745533 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745538 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745542 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745547 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745552 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745556 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745560 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745564 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745569 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745573 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745577 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745582 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745586 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745590 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:30.749077 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745594 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745599 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745603 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745607 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745611 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745615 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745621 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745628 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745635 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745640 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745646 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745650 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745654 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745659 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745663 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745667 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745671 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745675 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:30.749721 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745680 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745684 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745688 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745692 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745696 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745701 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745705 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745709 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745713 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745718 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745722 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745726 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745730 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745734 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745738 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745743 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745747 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745751 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745755 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745760 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745765 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:30.750144 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745769 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745773 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745777 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745781 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745786 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745790 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745794 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745799 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745803 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745807 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745811 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745816 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745820 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.745824 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746674 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746683 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746688 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746693 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746697 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:30.750662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746702 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746706 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746710 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746715 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746719 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746724 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746728 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746732 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746737 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746742 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746747 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746751 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746755 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746760 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746764 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746768 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746772 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746776 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746781 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:30.751115 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746785 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746789 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746795 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746801 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746806 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746810 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746814 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746819 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746823 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746827 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746831 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746836 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746840 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746844 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746848 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746853 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746857 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746861 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746865 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:30.751574 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746870 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746874 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746879 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746883 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746887 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746892 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746897 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746901 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746905 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746911 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746917 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746923 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746928 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746933 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746938 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746945 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746950 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746954 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746959 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:30.752031 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746963 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746968 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746972 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746976 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746980 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746984 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746989 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746993 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.746998 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747002 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747007 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747011 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747016 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747021 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747025 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747029 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747033 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747037 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747042 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747046 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:30.752494 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747050 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747054 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747058 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747063 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747067 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747071 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747075 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747079 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747083 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747087 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747091 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747095 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747099 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747103 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747107 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747115 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747119 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747123 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747127 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747132 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:30.752955 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747136 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747140 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747145 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.747152 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747856 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747871 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747881 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747889 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747896 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747901 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747907 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747914 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747920 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747925 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747930 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747936 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747941 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747945 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747950 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747954 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747959 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747972 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747980 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747988 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:16:30.753421 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747993 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.747998 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748003 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748009 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748015 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748020 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748025 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748029 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748035 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748040 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748045 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748052 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748059 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748064 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748069 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748074 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748080 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748084 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748091 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748096 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748101 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748106 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748111 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748122 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:16:30.753972 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748127 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748132 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748137 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748142 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748147 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748151 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748156 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748162 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748167 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748172 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748178 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748183 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748188 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748193 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748199 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748204 2578 flags.go:64] FLAG: --help="false" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748209 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-134-254.ec2.internal" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.748266 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749628 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749786 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749797 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749803 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749807 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:16:30.754620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749810 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749813 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749816 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749820 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749823 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749826 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749830 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749833 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749836 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749839 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749842 2578 flags.go:64] FLAG: --lock-file="" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749845 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749849 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749852 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749859 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749862 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749865 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749868 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749871 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749875 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749878 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749881 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749886 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749891 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749895 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 01:16:30.755168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749898 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749902 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749905 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749908 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749912 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749915 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749918 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749936 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749939 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749944 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749948 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749951 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749954 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749957 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749960 2578 flags.go:64] FLAG: --port="10250" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749963 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749966 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01aab64cde907bb97" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749969 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749972 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749975 2578 flags.go:64] FLAG: --register-node="true" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749978 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749982 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749985 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749988 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749990 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749994 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:16:30.755792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.749998 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750001 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750004 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750007 2578 flags.go:64] FLAG: --runonce="false" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750010 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750013 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750016 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750020 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750023 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750026 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750029 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750033 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750036 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750039 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750041 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750044 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750047 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750050 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750054 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750060 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750063 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750066 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750071 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750074 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750077 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:16:30.756451 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750080 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750083 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750086 2578 flags.go:64] FLAG: --v="2" Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750090 2578 flags.go:64] FLAG: --version="false" Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750095 2578 flags.go:64] FLAG: --vmodule="" Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750099 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750103 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750244 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750248 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750252 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750256 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750259 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750264 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750267 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750270 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750273 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750276 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750279 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750282 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750284 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750287 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:30.757046 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750290 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750292 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750295 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750297 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750300 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750302 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750305 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750308 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750310 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750313 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750316 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750318 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750321 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750339 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750342 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750345 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750348 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750350 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750353 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:30.757559 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750355 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750358 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750360 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750363 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750366 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750369 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750371 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750374 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750377 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750380 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750382 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750385 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750387 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750390 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750393 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750395 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750398 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750400 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750403 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750405 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:30.758003 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750408 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750410 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750413 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750415 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750418 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750421 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750423 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750426 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750429 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750432 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750434 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750437 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750439 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750443 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750446 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750449 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750452 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750455 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750458 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:30.758536 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750461 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750463 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750466 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750469 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750477 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750480 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750483 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750486 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750488 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750491 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750494 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750496 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750499 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750502 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750504 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750507 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750509 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750512 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750514 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750517 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:30.759078 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750519 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750522 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750525 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750527 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750530 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750532 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750535 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750538 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750540 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750543 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750545 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750548 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750550 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750553 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.750580 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750621 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:30.759555 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750626 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750629 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750631 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750634 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750638 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750641 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750644 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750648 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750651 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750653 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750656 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750658 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750661 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750663 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750666 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750669 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750671 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750674 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750677 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:30.759943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750680 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750682 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750690 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750693 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750696 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750698 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750701 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750703 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750706 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750708 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750711 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750714 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750716 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750719 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750721 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750725 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750728 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750731 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750734 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:30.760397 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750737 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750740 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750742 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750745 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750747 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750750 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750753 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750755 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750758 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750761 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750763 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750766 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750768 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750771 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750775 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750778 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750781 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750783 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750786 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:30.760843 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750789 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750791 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750796 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750798 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750801 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750803 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750806 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750809 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750811 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750813 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750816 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750819 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750821 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750824 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750826 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750829 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750831 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750834 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750836 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750839 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:30.761297 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750842 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750844 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750847 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750849 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750852 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750854 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750857 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750860 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750863 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750866 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750869 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750872 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750874 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750877 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750884 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750886 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750889 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750891 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750895 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750899 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:30.761823 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750902 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750905 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750908 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750910 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750914 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750916 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750919 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.750922 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.757965 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.757983 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.757997 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.758002 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758060 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758065 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758069 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758072 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758075 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758077 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758080 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:30.762296 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758082 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758085 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758088 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758090 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758093 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758096 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758098 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758102 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758104 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758107 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758110 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758112 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758115 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758117 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758120 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758122 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758125 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758127 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758130 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:30.762754 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758132 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758136 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758140 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758145 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758148 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758151 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758154 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758156 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758159 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758161 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758164 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758167 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758169 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758172 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758174 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758177 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758179 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758182 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758185 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:30.763204 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758187 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758189 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758192 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758195 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758197 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758200 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758202 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758205 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758207 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758210 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758212 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758215 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758218 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758220 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758223 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758225 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758228 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758232 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758235 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758238 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:30.763786 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758240 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758243 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758245 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758248 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758251 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758253 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758256 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758258 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758261 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758264 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758266 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758269 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758271 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758274 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758276 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758280 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758285 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758287 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758290 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:30.764365 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758293 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758295 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758298 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758300 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758303 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758305 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758308 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758311 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758313 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758316 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758319 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758342 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758348 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758351 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758354 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758356 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758359 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758362 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758364 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758366 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:30.764804 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758369 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758372 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.758399 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758540 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758544 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758548 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758550 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758554 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758557 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758560 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758562 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758565 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758568 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758570 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758573 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758575 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:30.765350 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758578 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758580 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758583 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758586 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758588 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758592 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758594 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758597 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758600 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758603 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758606 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758608 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758611 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758614 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758617 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758619 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758622 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758624 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758626 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758629 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:30.765712 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758631 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758635 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758639 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758641 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758644 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758647 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758650 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758652 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758655 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758657 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758660 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758662 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758665 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758667 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758670 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758672 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758675 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758678 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758680 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:30.766190 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758682 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758685 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758688 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758691 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758693 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758696 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758698 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758702 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758705 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758708 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758712 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758715 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758717 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758720 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758723 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758726 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758729 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758731 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758734 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758737 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:30.766703 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758740 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758742 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758745 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758747 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758750 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758752 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758755 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758758 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758760 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758763 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758766 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758768 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758771 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758773 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758776 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758779 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758782 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758785 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758788 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758790 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:30.767201 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758792 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758795 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758797 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758800 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758802 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758805 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758807 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758810 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758812 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758815 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758817 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758820 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758822 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758825 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.758847 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758882 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:30.767726 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758886 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758889 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758892 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758894 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758897 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758899 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758902 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758905 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758907 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758911 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758915 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758918 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758922 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758924 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758927 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758930 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758932 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758934 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758937 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:30.768135 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758939 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758942 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758945 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758947 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758950 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758952 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758955 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758957 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758960 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758962 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758965 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758968 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758970 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758973 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758975 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758978 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758980 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758982 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758985 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758988 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:30.768758 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758991 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758994 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758996 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.758999 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759002 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759004 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759007 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759010 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759012 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759014 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759017 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759019 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759022 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759024 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759027 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759029 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759032 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759034 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759037 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759039 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:30.769243 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759041 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759044 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759047 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759049 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759052 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759054 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759057 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759059 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759062 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759064 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759067 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759069 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759072 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759074 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759077 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759079 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759082 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759085 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759088 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:30.769982 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759091 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759093 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759096 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759098 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759103 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759105 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759108 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759110 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759113 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759115 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759117 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759120 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759124 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759128 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759131 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759134 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759137 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759140 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759142 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:30.770718 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759146 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759149 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759152 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759154 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759157 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759159 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759162 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:30.759165 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.759222 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.761237 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.762166 2578 server.go:1014] "Starting client certificate rotation" Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.762259 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.762873 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.763931 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:16:30.771254 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.765069 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:16:30.774574 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.774557 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:16:30.778510 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.778495 2578 log.go:25] "Validated CRI v1 image API" Oct 30 01:16:30.779263 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.779237 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:16:30.781648 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.781631 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d0c94ca6-26bc-4b67-beab-cb2f8f648b10:/dev/nvme0n1p4 edd7a548-da53-49cb-adf7-188d93ff64a4:/dev/nvme0n1p3] Oct 30 01:16:30.781705 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.781649 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 01:16:30.786580 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.786471 2578 manager.go:217] Machine: {Timestamp:2026-10-30 01:16:30.785981558 +0000 UTC m=+0.324322593 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099732 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e83693b95fc7e2d31cdc46e86d507 SystemUUID:ec2e8369-3b95-fc7e-2d31-cdc46e86d507 BootID:1ada4c04-365c-4a45-b0ba-61eba918e8e4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d4:42:d9:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d4:42:d9:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:8f:ef:61:bb:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:16:30.787167 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.787156 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:16:30.787282 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.787263 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:16:30.788371 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.788350 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:16:30.788542 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.788372 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-254.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:16:30.788588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.788546 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:16:30.788588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.788554 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:16:30.788588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.788571 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:16:30.788588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.788585 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:16:30.789394 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.789385 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:16:30.792080 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.792068 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:16:30.792201 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.792192 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:16:30.792790 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.792781 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:16:30.792827 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.792794 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:16:30.792827 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.792803 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:16:30.792827 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.792811 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:16:30.792827 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.792819 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:16:30.793132 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.793114 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:16:30.795879 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.795864 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:16:30.800572 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.800559 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:16:30.801302 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.801292 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:16:30.801355 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.801321 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:16:30.801510 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.801500 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:16:30.801544 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.801517 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:16:30.801544 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.801523 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:16:30.801544 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.801528 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:16:30.801544 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.801534 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:16:30.801544 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.801543 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:16:30.802048 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.802040 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:16:30.802076 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.802049 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:16:30.802076 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.802056 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:16:30.802076 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.802062 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:16:30.802076 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.802070 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:16:30.802230 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.802079 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:16:30.802723 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.802713 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:16:30.802761 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.802725 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:16:30.806419 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.806408 2578 server.go:1263] "Started kubelet" Oct 30 01:16:30.806672 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.806611 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:16:30.806783 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.806681 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:16:30.807033 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.807003 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:16:30.807101 ip-10-0-134-254 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:16:30.807639 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.807626 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:16:30.810668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.810647 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:16:30.814556 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.814522 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 01:16:30.815369 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.815352 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:16:30.815453 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.815387 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:16:30.815453 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.815429 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:30.815958 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.815924 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:16:30.815958 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.815947 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:16:30.816095 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.815921 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:16:30.816095 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.816045 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:16:30.816095 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.816052 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:16:30.816258 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.816241 2578 factory.go:55] Registering systemd factory Oct 30 01:16:30.816258 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.816257 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:16:30.816557 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.816479 2578 factory.go:153] Registering CRI-O factory Oct 30 01:16:30.816557 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.816495 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 01:16:30.816557 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.816547 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:16:30.816674 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.816563 2578 factory.go:103] Registering Raw factory Oct 30 01:16:30.816674 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.816575 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 01:16:30.817778 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.817751 2578 manager.go:319] Starting recovery of all containers Oct 30 01:16:30.825225 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.825195 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:16:30.826223 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.826206 2578 manager.go:324] Recovery completed Oct 30 01:16:30.831027 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.831014 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:30.833134 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.833110 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:30.833197 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.833148 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:30.833197 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.833158 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:30.833639 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.833625 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:16:30.833699 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.833640 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:16:30.833699 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.833655 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:30.833699 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.833659 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:16:30.834628 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.834615 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:30.834702 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.834640 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:30.834702 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.834651 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:30.835030 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.835019 2578 policy_none.go:50] "Start" Oct 30 01:16:30.835061 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.835034 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:16:30.835061 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.835044 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:16:30.836154 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.836144 2578 policy_none.go:44] "Start" Oct 30 01:16:30.838044 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.838019 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-254.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 01:16:30.838159 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.838131 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 01:16:30.841697 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.838144 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-254.ec2.internal.18e329c23a8e1ff0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-254.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-254.ec2.internal,},FirstTimestamp:2026-10-30 01:16:30.806376432 +0000 UTC m=+0.344717364,LastTimestamp:2026-10-30 01:16:30.806376432 +0000 UTC m=+0.344717364,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-254.ec2.internal,}" Oct 30 01:16:30.853707 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.853683 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 01:16:30.853707 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.853685 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 01:16:30.853806 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.853684 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-254.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 01:16:30.860364 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.860344 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:16:30.880745 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.880678 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-254.ec2.internal.18e329c23c2664ff default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-254.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-254.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-254.ec2.internal,},FirstTimestamp:2026-10-30 01:16:30.833132799 +0000 UTC m=+0.371473731,LastTimestamp:2026-10-30 01:16:30.833132799 +0000 UTC m=+0.371473731,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-254.ec2.internal,}" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.881041 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8wjl2" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.886092 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.889647 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.889675 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.889682 2578 server.go:87] "Starting device plugin registration server" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.889936 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.889947 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.890044 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8wjl2" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.890102 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.890456 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.890466 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.890608 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.890644 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.899011 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-254.ec2.internal.18e329c23c26b413 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-254.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-254.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-254.ec2.internal,},FirstTimestamp:2026-10-30 01:16:30.833153043 +0000 UTC m=+0.371493975,LastTimestamp:2026-10-30 01:16:30.833153043 +0000 UTC m=+0.371493975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-254.ec2.internal,}" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.900241 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.900261 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.900278 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:16:30.901074 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:30.900307 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 01:16:30.926362 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.926344 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:30.990172 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.990102 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:30.990981 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.990966 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:30.991034 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.990995 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:30.991034 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.991004 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:30.991034 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:30.991034 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.001129 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.001113 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal"] Oct 30 01:16:31.001198 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.001187 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:31.001957 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.001944 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:31.002021 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.001968 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:31.002021 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.001980 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:31.003621 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.003610 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:31.003793 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.003780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.003831 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.003808 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:31.004308 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.004294 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:31.004414 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.004314 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:31.004414 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.004344 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:31.004414 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.004391 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:31.004414 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.004414 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:31.004599 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.004429 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:31.005545 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.005531 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.005603 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.005563 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:31.006355 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.006339 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:31.006435 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.006371 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:31.006435 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.006384 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:31.008375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.008362 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.008430 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.008380 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-254.ec2.internal\": node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:31.032036 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.032012 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-254.ec2.internal\" not found" node="ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.035554 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.035536 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-254.ec2.internal\" not found" node="ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.064933 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.064916 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:31.117572 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.117547 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.117638 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.117571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.117638 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.117590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dddf377448d96e272c684d4e526ba4c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-254.ec2.internal\" (UID: \"7dddf377448d96e272c684d4e526ba4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.165937 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.165920 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:31.218715 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.218694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.218774 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.218719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.218774 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.218734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dddf377448d96e272c684d4e526ba4c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-254.ec2.internal\" (UID: \"7dddf377448d96e272c684d4e526ba4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.218835 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.218783 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dddf377448d96e272c684d4e526ba4c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-254.ec2.internal\" (UID: \"7dddf377448d96e272c684d4e526ba4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.218835 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.218789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.218835 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.218798 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.266116 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.266078 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:31.335274 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.335248 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.339439 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.339131 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.366917 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.366893 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:31.467308 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.467278 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:31.567755 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.567697 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:31.668358 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.668318 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:31.762008 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.761777 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 01:16:31.762478 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.762164 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:16:31.769163 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:31.769143 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:16:31.830347 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:31.830296 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dddf377448d96e272c684d4e526ba4c.slice/crio-cdd603789b0b0581f00e91ebf74be898007d44ba6a48597712bde59946488b93 WatchSource:0}: Error finding container cdd603789b0b0581f00e91ebf74be898007d44ba6a48597712bde59946488b93: Status 404 returned error can't find the container with id cdd603789b0b0581f00e91ebf74be898007d44ba6a48597712bde59946488b93 Oct 30 01:16:31.834738 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.834720 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:16:31.842049 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.842029 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:31.850662 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:31.850642 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f52b57e04740fa6edc4f85ae11f68f.slice/crio-015b60aeb8a694313ddfb327651057a6e9449ec69363a98c0b78412de46af90c WatchSource:0}: Error finding container 015b60aeb8a694313ddfb327651057a6e9449ec69363a98c0b78412de46af90c: Status 404 returned error can't find the container with id 015b60aeb8a694313ddfb327651057a6e9449ec69363a98c0b78412de46af90c Oct 30 01:16:31.894541 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.894511 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:11:31 +0000 UTC" deadline="2028-04-22 00:33:59.561629574 +0000 UTC" Oct 30 01:16:31.894541 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.894536 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12959h17m27.667096s" Oct 30 01:16:31.903623 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.903585 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" event={"ID":"26f52b57e04740fa6edc4f85ae11f68f","Type":"ContainerStarted","Data":"015b60aeb8a694313ddfb327651057a6e9449ec69363a98c0b78412de46af90c"} Oct 30 01:16:31.904662 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.904639 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" event={"ID":"7dddf377448d96e272c684d4e526ba4c","Type":"ContainerStarted","Data":"cdd603789b0b0581f00e91ebf74be898007d44ba6a48597712bde59946488b93"} Oct 30 01:16:31.915671 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.915657 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.941946 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.941930 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:16:31.943781 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.943768 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:16:31.959288 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.959274 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:31.965269 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:31.965253 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:16:32.049775 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.049747 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:32.793344 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.793299 2578 apiserver.go:52] "Watching apiserver" Oct 30 01:16:32.816892 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.816862 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 01:16:32.845436 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.845401 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:16:32.845884 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.845860 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-vx2b8","kube-system/konnectivity-agent-mvmkl","kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal","openshift-cluster-node-tuning-operator/tuned-f4chz","openshift-image-registry/node-ca-kckpv","openshift-network-diagnostics/network-check-target-4cnv8","openshift-dns/node-resolver-9gckm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal","openshift-multus/network-metrics-daemon-p9qrg","kube-system/global-pull-secret-syncer-rz57q","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg","openshift-multus/multus-nvmrz","openshift-multus/multus-additional-cni-plugins-v4slr","openshift-network-operator/iptables-alerter-blx29"] Oct 30 01:16:32.847531 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.847507 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:32.849493 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.849466 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:32.851134 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.851111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:32.852539 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.852516 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.853994 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.853962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:32.855430 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.855405 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:32.855525 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:32.855481 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:32.857190 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.857174 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:32.857273 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:32.857256 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:32.858524 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.858506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.860027 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.860004 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:32.861707 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.861684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:32.861806 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:32.861737 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:32.863601 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.863311 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:32.865117 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.865096 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.866045 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.866026 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pcscp\"" Oct 30 01:16:32.866308 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.866290 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:16:32.866391 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.866337 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:16:32.866996 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.866980 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:16:32.884640 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.884615 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:16:32.884741 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.884690 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-d2dgz" Oct 30 01:16:32.917397 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.917376 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:16:32.926401 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.926378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f207f3a7-cf1c-4acc-8847-267e7c89e31f-host-slash\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:32.926601 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.926574 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.926718 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.926613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-config\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.926718 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.926639 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-modprobe-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.926718 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.926678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-k8s-cni-cncf-io\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.926891 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.926731 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-multus\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.926891 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.926791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-kubelet\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.926997 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.926890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-env-overrides\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.926997 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.926917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:32.926997 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.926974 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwd46\" (UniqueName: \"kubernetes.io/projected/d8c54d89-eea4-45ee-80e5-fd0aba53e274-kube-api-access-nwd46\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:32.927153 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927001 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-var-lib-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.927153 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:32.927153 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927052 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-socket-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:32.927153 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-registration-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:32.927153 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927100 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-device-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:32.927153 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927135 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/628af330-6652-4df0-8275-9f691e2c8eb3-konnectivity-ca\") pod \"konnectivity-agent-mvmkl\" (UID: \"628af330-6652-4df0-8275-9f691e2c8eb3\") " pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927160 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cnibin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p5mm\" (UniqueName: \"kubernetes.io/projected/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-kube-api-access-9p5mm\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-script-lib\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-lib-modules\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-system-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927340 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927387 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-etc-tuned\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927431 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/628af330-6652-4df0-8275-9f691e2c8eb3-agent-certs\") pod \"konnectivity-agent-mvmkl\" (UID: \"628af330-6652-4df0-8275-9f691e2c8eb3\") " pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-cnibin\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927480 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-var-lib-kubelet\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927527 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-host\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.927546 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-conf-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927577 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovn-node-metrics-cert\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927603 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927628 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-dbus\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927716 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b47f8e4a-5060-47bf-8bec-722ccff0fa51-hosts-file\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927741 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-bin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927780 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqbqr\" (UniqueName: \"kubernetes.io/projected/335e4ad1-7154-4e2c-a170-d52aee2df045-kube-api-access-dqbqr\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25s9d\" (UniqueName: \"kubernetes.io/projected/f207f3a7-cf1c-4acc-8847-267e7c89e31f-kube-api-access-25s9d\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927832 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-netns\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927859 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-kubelet-config\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-systemd\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927913 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927940 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-ovn\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.928322 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.927991 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-conf\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-netns\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-kubelet\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928081 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-system-cni-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928120 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-log-socket\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzcm2\" (UniqueName: \"kubernetes.io/projected/97b353f9-eeeb-4f6a-bdb7-204450f76167-kube-api-access-bzcm2\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928206 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysconfig\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-daemon-config\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928278 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-multus-certs\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928304 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-etc-kubernetes\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-node-log\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-bin\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928378 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-kubernetes\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-hostroot\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928445 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29vzs\" (UniqueName: \"kubernetes.io/projected/3db07759-c867-4de9-84c7-24d903510dca-kube-api-access-29vzs\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.928960 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928470 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9801105-8f8a-45d0-b2c1-23468a212347-serviceca\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928495 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trgqt\" (UniqueName: \"kubernetes.io/projected/c9801105-8f8a-45d0-b2c1-23468a212347-kube-api-access-trgqt\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cni-binary-copy\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928571 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-systemd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928736 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-socket-dir-parent\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928758 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f207f3a7-cf1c-4acc-8847-267e7c89e31f-iptables-alerter-script\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928775 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-slash\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-etc-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928862 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-tmp\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928920 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-systemd-units\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-netd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.928987 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2529z\" (UniqueName: \"kubernetes.io/projected/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-kube-api-access-2529z\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.929015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-sys-fs\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.929056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-run\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.929098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-sys\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.929132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47f8e4a-5060-47bf-8bec-722ccff0fa51-tmp-dir\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.929161 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrqvr\" (UniqueName: \"kubernetes.io/projected/b47f8e4a-5060-47bf-8bec-722ccff0fa51-kube-api-access-hrqvr\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:32.929499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.929193 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9801105-8f8a-45d0-b2c1-23468a212347-host\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:32.975680 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.975578 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:16:32.975922 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.975877 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:16:32.976038 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.975961 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:16:32.976106 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976041 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:16:32.976175 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976105 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:16:32.976175 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976112 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:16:32.976175 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976127 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-58lhf\"" Oct 30 01:16:32.976323 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976177 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:16:32.976323 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976200 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fsgft\"" Oct 30 01:16:32.976323 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976209 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p4hjp\"" Oct 30 01:16:32.976323 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976220 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:16:32.976323 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976266 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2b59\"" Oct 30 01:16:32.976323 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976282 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976356 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976363 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976380 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976437 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976447 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976449 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976492 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976497 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976531 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8bsj\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976497 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:16:32.976622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.976499 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:16:32.985156 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.985130 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hb7tb\"" Oct 30 01:16:32.991109 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.991086 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:16:32.991600 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.991533 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:16:32.991600 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.991542 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:16:32.991600 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.991540 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-srg4h\"" Oct 30 01:16:32.991600 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.991564 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:16:32.991860 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.991613 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7dsj7\"" Oct 30 01:16:32.992312 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:32.992289 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:16:33.030295 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vzs\" (UniqueName: \"kubernetes.io/projected/3db07759-c867-4de9-84c7-24d903510dca-kube-api-access-29vzs\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.030419 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9801105-8f8a-45d0-b2c1-23468a212347-serviceca\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:33.030419 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030325 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trgqt\" (UniqueName: \"kubernetes.io/projected/c9801105-8f8a-45d0-b2c1-23468a212347-kube-api-access-trgqt\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:33.030419 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cni-binary-copy\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.030419 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030389 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-systemd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.030419 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030407 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-socket-dir-parent\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f207f3a7-cf1c-4acc-8847-267e7c89e31f-iptables-alerter-script\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-slash\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-etc-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-tmp\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-systemd-units\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-netd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2529z\" (UniqueName: \"kubernetes.io/projected/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-kube-api-access-2529z\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030576 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-sys-fs\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-run\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030610 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-sys\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030625 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47f8e4a-5060-47bf-8bec-722ccff0fa51-tmp-dir\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqvr\" (UniqueName: \"kubernetes.io/projected/b47f8e4a-5060-47bf-8bec-722ccff0fa51-kube-api-access-hrqvr\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030653 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9801105-8f8a-45d0-b2c1-23468a212347-host\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:33.030668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030674 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f207f3a7-cf1c-4acc-8847-267e7c89e31f-host-slash\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030727 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-config\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-modprobe-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-k8s-cni-cncf-io\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030795 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-multus\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-kubelet\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030825 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-env-overrides\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030853 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030878 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwd46\" (UniqueName: \"kubernetes.io/projected/d8c54d89-eea4-45ee-80e5-fd0aba53e274-kube-api-access-nwd46\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030907 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-var-lib-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030975 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-var-lib-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.031046 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.031114 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f207f3a7-cf1c-4acc-8847-267e7c89e31f-host-slash\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:33.031289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.031168 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.031886 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.030669 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-slash\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.031886 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.031616 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9801105-8f8a-45d0-b2c1-23468a212347-host\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:33.031886 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.031664 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-etc-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.032036 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.031888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9801105-8f8a-45d0-b2c1-23468a212347-serviceca\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:33.032036 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.031928 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-config\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.032036 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.031982 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:16:33.032144 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032045 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-modprobe-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.032144 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032085 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-k8s-cni-cncf-io\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.032144 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032123 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-multus\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.032258 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032158 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-kubelet\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.032258 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032190 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-systemd-units\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.032330 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.032285 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:33.032385 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.032333 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret podName:8bc436c0-2b1b-477e-81aa-af1a1eff9779 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:33.53231842 +0000 UTC m=+3.070659362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret") pod "global-pull-secret-syncer-rz57q" (UID: "8bc436c0-2b1b-477e-81aa-af1a1eff9779") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:33.032572 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032548 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cni-binary-copy\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.032640 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032624 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-netd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.032764 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032718 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.032822 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032765 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f207f3a7-cf1c-4acc-8847-267e7c89e31f-iptables-alerter-script\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:33.032926 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032903 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-env-overrides\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.032987 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.032928 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-systemd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.033036 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033004 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-sys\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.033080 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033067 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-socket-dir-parent\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.033140 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-socket-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.033185 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033168 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-sys-fs\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.033284 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-registration-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.033284 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033256 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-device-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.033378 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033299 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47f8e4a-5060-47bf-8bec-722ccff0fa51-tmp-dir\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:33.033378 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033327 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-socket-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.033378 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033349 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-registration-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.033378 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033371 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-device-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.033563 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033386 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-run\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.033563 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/628af330-6652-4df0-8275-9f691e2c8eb3-konnectivity-ca\") pod \"konnectivity-agent-mvmkl\" (UID: \"628af330-6652-4df0-8275-9f691e2c8eb3\") " pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:33.033563 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cnibin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.033563 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p5mm\" (UniqueName: \"kubernetes.io/projected/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-kube-api-access-9p5mm\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.033563 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.033563 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-script-lib\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033563 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033568 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cnibin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-lib-modules\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-system-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033639 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-etc-tuned\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/628af330-6652-4df0-8275-9f691e2c8eb3-agent-certs\") pod \"konnectivity-agent-mvmkl\" (UID: \"628af330-6652-4df0-8275-9f691e2c8eb3\") " pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033737 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-cnibin\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-var-lib-kubelet\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033815 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-host\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-conf-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033862 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovn-node-metrics-cert\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.034026 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033932 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/628af330-6652-4df0-8275-9f691e2c8eb3-konnectivity-ca\") pod \"konnectivity-agent-mvmkl\" (UID: \"628af330-6652-4df0-8275-9f691e2c8eb3\") " pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033979 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.033981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-dbus\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b47f8e4a-5060-47bf-8bec-722ccff0fa51-hosts-file\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-bin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqbqr\" (UniqueName: \"kubernetes.io/projected/335e4ad1-7154-4e2c-a170-d52aee2df045-kube-api-access-dqbqr\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034106 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-dbus\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034109 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25s9d\" (UniqueName: \"kubernetes.io/projected/f207f3a7-cf1c-4acc-8847-267e7c89e31f-kube-api-access-25s9d\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034137 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-netns\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-kubelet-config\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034186 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-systemd\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034223 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-ovn\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034311 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-conf\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.034853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-netns\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-kubelet\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034386 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-system-cni-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034423 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-log-socket\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034464 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034490 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-host\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034492 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzcm2\" (UniqueName: \"kubernetes.io/projected/97b353f9-eeeb-4f6a-bdb7-204450f76167-kube-api-access-bzcm2\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysconfig\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-daemon-config\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-multus-certs\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034613 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-etc-kubernetes\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-node-log\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034635 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034665 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-bin\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034690 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-kubernetes\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034715 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-ovn\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-hostroot\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.035684 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034757 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-hostroot\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034823 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-conf-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034869 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-conf\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034916 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-netns\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034957 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-kubelet\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.034998 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-system-cni-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-log-socket\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-var-lib-kubelet\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035109 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-script-lib\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-systemd\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035253 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035294 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035388 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-system-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035461 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b47f8e4a-5060-47bf-8bec-722ccff0fa51-hosts-file\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-bin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.035643 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.035697 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs podName:d8c54d89-eea4-45ee-80e5-fd0aba53e274 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:33.535680765 +0000 UTC m=+3.074021701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs") pod "network-metrics-daemon-p9qrg" (UID: "d8c54d89-eea4-45ee-80e5-fd0aba53e274") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:33.036588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysconfig\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.035973 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-multus-certs\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036030 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-etc-kubernetes\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036075 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-bin\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-kubernetes\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036187 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-kubelet-config\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036233 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-netns\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036316 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-lib-modules\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036353 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-node-log\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036405 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036455 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-cnibin\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.037501 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.036771 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-daemon-config\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.038117 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.037722 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovn-node-metrics-cert\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.039317 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.039136 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-etc-tuned\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.039704 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.039659 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/628af330-6652-4df0-8275-9f691e2c8eb3-agent-certs\") pod \"konnectivity-agent-mvmkl\" (UID: \"628af330-6652-4df0-8275-9f691e2c8eb3\") " pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:33.045793 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.045758 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-tmp\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.055576 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.055549 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vzs\" (UniqueName: \"kubernetes.io/projected/3db07759-c867-4de9-84c7-24d903510dca-kube-api-access-29vzs\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.055994 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.055975 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:33.056107 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.055999 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:33.056107 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.056013 2578 projected.go:196] Error preparing data for projected volume kube-api-access-qq8g2 for pod openshift-network-diagnostics/network-check-target-4cnv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:33.056107 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.056065 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2 podName:dbf09128-a8c1-4d32-958b-cbd01b892097 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:33.556047863 +0000 UTC m=+3.094388809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qq8g2" (UniqueName: "kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2") pod "network-check-target-4cnv8" (UID: "dbf09128-a8c1-4d32-958b-cbd01b892097") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:33.059166 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.058544 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgqt\" (UniqueName: \"kubernetes.io/projected/c9801105-8f8a-45d0-b2c1-23468a212347-kube-api-access-trgqt\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:33.063835 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.060155 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p5mm\" (UniqueName: \"kubernetes.io/projected/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-kube-api-access-9p5mm\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.063835 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.060231 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwd46\" (UniqueName: \"kubernetes.io/projected/d8c54d89-eea4-45ee-80e5-fd0aba53e274-kube-api-access-nwd46\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:33.063835 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.063595 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25s9d\" (UniqueName: \"kubernetes.io/projected/f207f3a7-cf1c-4acc-8847-267e7c89e31f-kube-api-access-25s9d\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:33.063835 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.063602 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2529z\" (UniqueName: \"kubernetes.io/projected/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-kube-api-access-2529z\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.063835 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.063691 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzcm2\" (UniqueName: \"kubernetes.io/projected/97b353f9-eeeb-4f6a-bdb7-204450f76167-kube-api-access-bzcm2\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.064242 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.064217 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqvr\" (UniqueName: \"kubernetes.io/projected/b47f8e4a-5060-47bf-8bec-722ccff0fa51-kube-api-access-hrqvr\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:33.065354 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.065310 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqbqr\" (UniqueName: \"kubernetes.io/projected/335e4ad1-7154-4e2c-a170-d52aee2df045-kube-api-access-dqbqr\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.162066 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.162020 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9gckm" Oct 30 01:16:33.168825 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.168716 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:33.171539 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:33.171516 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47f8e4a_5060_47bf_8bec_722ccff0fa51.slice/crio-604b48f2a6bf900a68d28b8a4f655902823a2c35093a4b53e5392cf9d0abfe90 WatchSource:0}: Error finding container 604b48f2a6bf900a68d28b8a4f655902823a2c35093a4b53e5392cf9d0abfe90: Status 404 returned error can't find the container with id 604b48f2a6bf900a68d28b8a4f655902823a2c35093a4b53e5392cf9d0abfe90 Oct 30 01:16:33.180743 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.180222 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:16:33.183297 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.183280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:16:33.189597 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.189570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:16:33.192573 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:33.192532 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335e4ad1_7154_4e2c_a170_d52aee2df045.slice/crio-967743bc57a5fffc01f1ec8c160c9dba25d32d19af6bfa5ac75eff37ceb93c6b WatchSource:0}: Error finding container 967743bc57a5fffc01f1ec8c160c9dba25d32d19af6bfa5ac75eff37ceb93c6b: Status 404 returned error can't find the container with id 967743bc57a5fffc01f1ec8c160c9dba25d32d19af6bfa5ac75eff37ceb93c6b Oct 30 01:16:33.196176 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:33.196132 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf207f3a7_cf1c_4acc_8847_267e7c89e31f.slice/crio-bc10f8646379b177bae48b25a694c1ed7902e21be6b60df1c2af0a3f5651a57a WatchSource:0}: Error finding container bc10f8646379b177bae48b25a694c1ed7902e21be6b60df1c2af0a3f5651a57a: Status 404 returned error can't find the container with id bc10f8646379b177bae48b25a694c1ed7902e21be6b60df1c2af0a3f5651a57a Oct 30 01:16:33.199065 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.199046 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:16:33.199687 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:33.199628 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db07759_c867_4de9_84c7_24d903510dca.slice/crio-6f033c2b5c538efd7561ab40e91824e4d695d563384594a0f61ff5f87bdd0348 WatchSource:0}: Error finding container 6f033c2b5c538efd7561ab40e91824e4d695d563384594a0f61ff5f87bdd0348: Status 404 returned error can't find the container with id 6f033c2b5c538efd7561ab40e91824e4d695d563384594a0f61ff5f87bdd0348 Oct 30 01:16:33.211564 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:33.211530 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9801105_8f8a_45d0_b2c1_23468a212347.slice/crio-c608864e5122f9ff1fe5c5d9696eabf37d485e59a24c65682c6d99bc62ec7be2 WatchSource:0}: Error finding container c608864e5122f9ff1fe5c5d9696eabf37d485e59a24c65682c6d99bc62ec7be2: Status 404 returned error can't find the container with id c608864e5122f9ff1fe5c5d9696eabf37d485e59a24c65682c6d99bc62ec7be2 Oct 30 01:16:33.219223 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.219202 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:33.220057 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.219994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:33.228885 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.228865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:16:33.229713 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:33.229676 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b388420_2ed8_4ee3_9f1f_899e2b8ae0c3.slice/crio-38368dbe1ec9a310be611eb35869a106c185febe1e2f4d4f9ca7eb5375bbf6e9 WatchSource:0}: Error finding container 38368dbe1ec9a310be611eb35869a106c185febe1e2f4d4f9ca7eb5375bbf6e9: Status 404 returned error can't find the container with id 38368dbe1ec9a310be611eb35869a106c185febe1e2f4d4f9ca7eb5375bbf6e9 Oct 30 01:16:33.233113 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:33.233075 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628af330_6652_4df0_8275_9f691e2c8eb3.slice/crio-fad6384529a2299251911a6192babdeea6fe6065a4c8b63c516bffc1d9a495d7 WatchSource:0}: Error finding container fad6384529a2299251911a6192babdeea6fe6065a4c8b63c516bffc1d9a495d7: Status 404 returned error can't find the container with id fad6384529a2299251911a6192babdeea6fe6065a4c8b63c516bffc1d9a495d7 Oct 30 01:16:33.233280 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.233217 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nvmrz" Oct 30 01:16:33.238867 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:33.238837 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b353f9_eeeb_4f6a_bdb7_204450f76167.slice/crio-7e79aac7069a84bc8f2b06cbd90ffb81ac70e134d00bf0bf268bd2b44b4b7b5d WatchSource:0}: Error finding container 7e79aac7069a84bc8f2b06cbd90ffb81ac70e134d00bf0bf268bd2b44b4b7b5d: Status 404 returned error can't find the container with id 7e79aac7069a84bc8f2b06cbd90ffb81ac70e134d00bf0bf268bd2b44b4b7b5d Oct 30 01:16:33.244441 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:16:33.244417 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4beb5e05_bee4_4eb3_b205_b4e6bc5d9b17.slice/crio-1fbeea87e508fddc4b3b851b7144df573860432e5448d5bc8341f7d01362aa59 WatchSource:0}: Error finding container 1fbeea87e508fddc4b3b851b7144df573860432e5448d5bc8341f7d01362aa59: Status 404 returned error can't find the container with id 1fbeea87e508fddc4b3b851b7144df573860432e5448d5bc8341f7d01362aa59 Oct 30 01:16:33.537789 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.537711 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:33.537789 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.537765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:33.537976 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.537874 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:33.537976 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.537876 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:33.537976 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.537940 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs podName:d8c54d89-eea4-45ee-80e5-fd0aba53e274 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:34.537921268 +0000 UTC m=+4.076262211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs") pod "network-metrics-daemon-p9qrg" (UID: "d8c54d89-eea4-45ee-80e5-fd0aba53e274") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:33.537976 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.537958 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret podName:8bc436c0-2b1b-477e-81aa-af1a1eff9779 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:34.537950912 +0000 UTC m=+4.076291840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret") pod "global-pull-secret-syncer-rz57q" (UID: "8bc436c0-2b1b-477e-81aa-af1a1eff9779") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:33.639227 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.639169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:33.639417 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.639322 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:33.639417 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.639340 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:33.639417 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.639353 2578 projected.go:196] Error preparing data for projected volume kube-api-access-qq8g2 for pod openshift-network-diagnostics/network-check-target-4cnv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:33.639572 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.639433 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2 podName:dbf09128-a8c1-4d32-958b-cbd01b892097 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:34.639415425 +0000 UTC m=+4.177756348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qq8g2" (UniqueName: "kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2") pod "network-check-target-4cnv8" (UID: "dbf09128-a8c1-4d32-958b-cbd01b892097") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:33.887192 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.887078 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:11:33 +0000 UTC" deadline="2028-04-13 23:27:16.322177997 +0000 UTC" Oct 30 01:16:33.887192 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.887147 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12766h10m42.435035987s" Oct 30 01:16:33.901351 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.900817 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:33.901351 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:33.900957 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:33.923336 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.923302 2578 generic.go:356] "Generic (PLEG): container finished" podID="26f52b57e04740fa6edc4f85ae11f68f" containerID="a38bfbd5a97bb05a3b901a2adb00f175951d9bf279d4f773ef345d9528825b38" exitCode=0 Oct 30 01:16:33.923755 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.923729 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" event={"ID":"26f52b57e04740fa6edc4f85ae11f68f","Type":"ContainerDied","Data":"a38bfbd5a97bb05a3b901a2adb00f175951d9bf279d4f773ef345d9528825b38"} Oct 30 01:16:33.931275 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.931199 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nvmrz" event={"ID":"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17","Type":"ContainerStarted","Data":"1fbeea87e508fddc4b3b851b7144df573860432e5448d5bc8341f7d01362aa59"} Oct 30 01:16:33.933860 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.933795 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mvmkl" event={"ID":"628af330-6652-4df0-8275-9f691e2c8eb3","Type":"ContainerStarted","Data":"fad6384529a2299251911a6192babdeea6fe6065a4c8b63c516bffc1d9a495d7"} Oct 30 01:16:33.939028 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.938893 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" event={"ID":"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3","Type":"ContainerStarted","Data":"38368dbe1ec9a310be611eb35869a106c185febe1e2f4d4f9ca7eb5375bbf6e9"} Oct 30 01:16:33.944064 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.944034 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kckpv" event={"ID":"c9801105-8f8a-45d0-b2c1-23468a212347","Type":"ContainerStarted","Data":"c608864e5122f9ff1fe5c5d9696eabf37d485e59a24c65682c6d99bc62ec7be2"} Oct 30 01:16:33.973001 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.972971 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-blx29" event={"ID":"f207f3a7-cf1c-4acc-8847-267e7c89e31f","Type":"ContainerStarted","Data":"bc10f8646379b177bae48b25a694c1ed7902e21be6b60df1c2af0a3f5651a57a"} Oct 30 01:16:33.982797 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:33.982772 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" event={"ID":"7dddf377448d96e272c684d4e526ba4c","Type":"ContainerStarted","Data":"56ae7997392d347539587e80d4f4dd8d876f3782548c1b534cb1d7d53c68d06a"} Oct 30 01:16:34.004964 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.004888 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" event={"ID":"97b353f9-eeeb-4f6a-bdb7-204450f76167","Type":"ContainerStarted","Data":"7e79aac7069a84bc8f2b06cbd90ffb81ac70e134d00bf0bf268bd2b44b4b7b5d"} Oct 30 01:16:34.010063 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.010007 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f4chz" event={"ID":"3db07759-c867-4de9-84c7-24d903510dca","Type":"ContainerStarted","Data":"6f033c2b5c538efd7561ab40e91824e4d695d563384594a0f61ff5f87bdd0348"} Oct 30 01:16:34.029221 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.029128 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4slr" event={"ID":"335e4ad1-7154-4e2c-a170-d52aee2df045","Type":"ContainerStarted","Data":"967743bc57a5fffc01f1ec8c160c9dba25d32d19af6bfa5ac75eff37ceb93c6b"} Oct 30 01:16:34.041582 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.041547 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9gckm" event={"ID":"b47f8e4a-5060-47bf-8bec-722ccff0fa51","Type":"ContainerStarted","Data":"604b48f2a6bf900a68d28b8a4f655902823a2c35093a4b53e5392cf9d0abfe90"} Oct 30 01:16:34.546446 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.546151 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:34.546623 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.546511 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:34.546701 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:34.546645 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:34.546751 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:34.546708 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret podName:8bc436c0-2b1b-477e-81aa-af1a1eff9779 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:36.546687905 +0000 UTC m=+6.085028846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret") pod "global-pull-secret-syncer-rz57q" (UID: "8bc436c0-2b1b-477e-81aa-af1a1eff9779") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:34.547171 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:34.547137 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:34.547255 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:34.547193 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs podName:d8c54d89-eea4-45ee-80e5-fd0aba53e274 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:36.547176824 +0000 UTC m=+6.085517750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs") pod "network-metrics-daemon-p9qrg" (UID: "d8c54d89-eea4-45ee-80e5-fd0aba53e274") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:34.647221 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.647177 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:34.647412 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:34.647342 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:34.647412 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:34.647398 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:34.647522 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:34.647417 2578 projected.go:196] Error preparing data for projected volume kube-api-access-qq8g2 for pod openshift-network-diagnostics/network-check-target-4cnv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:34.647522 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:34.647476 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2 podName:dbf09128-a8c1-4d32-958b-cbd01b892097 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:36.647458148 +0000 UTC m=+6.185799085 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qq8g2" (UniqueName: "kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2") pod "network-check-target-4cnv8" (UID: "dbf09128-a8c1-4d32-958b-cbd01b892097") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:34.888665 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.887389 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:11:33 +0000 UTC" deadline="2028-06-22 03:53:14.315595322 +0000 UTC" Oct 30 01:16:34.888665 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.887431 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14426h36m39.42816939s" Oct 30 01:16:34.903928 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.903898 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:34.904054 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:34.904029 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:34.904500 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:34.904478 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:34.904773 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:34.904583 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:35.057117 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:35.057032 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" event={"ID":"26f52b57e04740fa6edc4f85ae11f68f","Type":"ContainerStarted","Data":"b0e3f16e514da64dbaf702b37db6257fa91b16e75dc27bf20f003ca2bc805c8c"} Oct 30 01:16:35.104190 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:35.103575 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" podStartSLOduration=3.103548452 podStartE2EDuration="3.103548452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:34.029186943 +0000 UTC m=+3.567527891" watchObservedRunningTime="2026-10-30 01:16:35.103548452 +0000 UTC m=+4.641889400" Oct 30 01:16:35.901177 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:35.900653 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:35.901177 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:35.900803 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:36.566856 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:36.566771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:36.567036 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:36.566856 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:36.567036 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:36.567007 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:36.567148 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:36.567071 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret podName:8bc436c0-2b1b-477e-81aa-af1a1eff9779 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:40.567049031 +0000 UTC m=+10.105389982 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret") pod "global-pull-secret-syncer-rz57q" (UID: "8bc436c0-2b1b-477e-81aa-af1a1eff9779") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:36.567538 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:36.567514 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:36.567654 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:36.567575 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs podName:d8c54d89-eea4-45ee-80e5-fd0aba53e274 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:40.567558217 +0000 UTC m=+10.105899141 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs") pod "network-metrics-daemon-p9qrg" (UID: "d8c54d89-eea4-45ee-80e5-fd0aba53e274") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:36.667240 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:36.667200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:36.667444 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:36.667422 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:36.667532 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:36.667447 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:36.667532 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:36.667461 2578 projected.go:196] Error preparing data for projected volume kube-api-access-qq8g2 for pod openshift-network-diagnostics/network-check-target-4cnv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:36.667532 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:36.667515 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2 podName:dbf09128-a8c1-4d32-958b-cbd01b892097 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:40.667497164 +0000 UTC m=+10.205838101 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qq8g2" (UniqueName: "kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2") pod "network-check-target-4cnv8" (UID: "dbf09128-a8c1-4d32-958b-cbd01b892097") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:36.903852 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:36.903772 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:36.904300 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:36.903904 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:36.904300 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:36.904281 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:36.904437 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:36.904373 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:37.071409 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:37.071097 2578 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="12e01038f8fadcbb2def5a45bc19095d0ae4692f670b50ca0b7234c7552bafdb" exitCode=0 Oct 30 01:16:37.071409 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:37.071149 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4slr" event={"ID":"335e4ad1-7154-4e2c-a170-d52aee2df045","Type":"ContainerDied","Data":"12e01038f8fadcbb2def5a45bc19095d0ae4692f670b50ca0b7234c7552bafdb"} Oct 30 01:16:37.164263 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:37.164162 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" podStartSLOduration=5.164143675 podStartE2EDuration="5.164143675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:35.104122415 +0000 UTC m=+4.642463361" watchObservedRunningTime="2026-10-30 01:16:37.164143675 +0000 UTC m=+6.702484623" Oct 30 01:16:37.900866 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:37.900786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:37.901081 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:37.900938 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:38.076736 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:38.076695 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mvmkl" event={"ID":"628af330-6652-4df0-8275-9f691e2c8eb3","Type":"ContainerStarted","Data":"33b98d5f5ac4088db04f2be7ef368b39566b23a494a6a82168f97f3e294ff098"} Oct 30 01:16:38.123418 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:38.123355 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mvmkl" podStartSLOduration=2.887525446 podStartE2EDuration="7.123334789s" totalImagesPullingTime="4.235809343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:38.12298236 +0000 UTC m=+7.661323305" watchObservedRunningTime="2026-10-30 01:16:38.123334789 +0000 UTC m=+7.661675965" Oct 30 01:16:38.903558 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:38.903485 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:38.903720 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:38.903634 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:38.903924 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:38.903886 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:38.904043 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:38.904017 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:39.083760 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:39.083723 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-kckpv" event={"ID":"c9801105-8f8a-45d0-b2c1-23468a212347","Type":"ContainerStarted","Data":"7313ff519b444e92c214fde879af10f6d9e9e43efa7c46ffb23690c6f95bc585"} Oct 30 01:16:39.128967 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:39.128909 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-kckpv" podStartSLOduration=2.90417646 podStartE2EDuration="8.128887133s" totalImagesPullingTime="5.224710673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:39.12863757 +0000 UTC m=+8.666978516" watchObservedRunningTime="2026-10-30 01:16:39.128887133 +0000 UTC m=+8.667228080" Oct 30 01:16:39.901428 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:39.901329 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:39.901601 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:39.901498 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:40.100066 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:40.100029 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" event={"ID":"97b353f9-eeeb-4f6a-bdb7-204450f76167","Type":"ContainerStarted","Data":"403da42b57e3e99dfe71d8b56cf1eaf94ad74a1c6c1a6d6d5e5362c173e5ce4e"} Oct 30 01:16:40.601405 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:40.601363 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:40.601564 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:40.601434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:40.601634 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:40.601563 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:40.601634 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:40.601621 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret podName:8bc436c0-2b1b-477e-81aa-af1a1eff9779 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:48.601602902 +0000 UTC m=+18.139943839 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret") pod "global-pull-secret-syncer-rz57q" (UID: "8bc436c0-2b1b-477e-81aa-af1a1eff9779") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:40.602015 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:40.601993 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:40.602112 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:40.602058 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs podName:d8c54d89-eea4-45ee-80e5-fd0aba53e274 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:48.602042446 +0000 UTC m=+18.140383379 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs") pod "network-metrics-daemon-p9qrg" (UID: "d8c54d89-eea4-45ee-80e5-fd0aba53e274") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:40.702634 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:40.701998 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:40.702634 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:40.702191 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:40.702634 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:40.702209 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:40.702634 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:40.702220 2578 projected.go:196] Error preparing data for projected volume kube-api-access-qq8g2 for pod openshift-network-diagnostics/network-check-target-4cnv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:40.702634 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:40.702286 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2 podName:dbf09128-a8c1-4d32-958b-cbd01b892097 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:48.70226825 +0000 UTC m=+18.240609175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qq8g2" (UniqueName: "kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2") pod "network-check-target-4cnv8" (UID: "dbf09128-a8c1-4d32-958b-cbd01b892097") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:40.907563 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:40.907269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:40.907563 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:40.907489 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:40.907563 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:40.907506 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:40.908502 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:40.908420 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:41.055185 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:41.055164 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:16:41.107679 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:41.107641 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" event={"ID":"97b353f9-eeeb-4f6a-bdb7-204450f76167","Type":"ContainerStarted","Data":"27447fc00a6f2a10b4fdc1cacef577b00a5d6ba16e8c1a02e559bb7b9c344159"} Oct 30 01:16:41.900973 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:41.900895 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:41.901095 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:41.901044 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:41.902669 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:41.902565 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:16:41.055182812Z","UUID":"1f662c26-bb25-4b8c-98af-dee6c9ec419f","Handler":null,"Name":"","Endpoint":""} Oct 30 01:16:41.904365 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:41.904343 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:16:41.904464 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:41.904377 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:16:42.113408 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:42.113305 2578 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="b74fc4b33d2d5ad89222c0f3242597fd72c63adfe79bcede2ee4c63a86b73204" exitCode=0 Oct 30 01:16:42.113408 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:42.113367 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4slr" event={"ID":"335e4ad1-7154-4e2c-a170-d52aee2df045","Type":"ContainerDied","Data":"b74fc4b33d2d5ad89222c0f3242597fd72c63adfe79bcede2ee4c63a86b73204"} Oct 30 01:16:42.115797 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:42.115775 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9gckm" event={"ID":"b47f8e4a-5060-47bf-8bec-722ccff0fa51","Type":"ContainerStarted","Data":"c148edf01b41005b101a24ddeeddd21ad079b3e635c9f2a127a756797dcfb888"} Oct 30 01:16:42.118885 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:42.118470 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nvmrz" event={"ID":"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17","Type":"ContainerStarted","Data":"60478c00d7369532265eaae35ab1c91cc8363bda063984a014da19cc3dbb88bd"} Oct 30 01:16:42.247480 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:42.246646 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9gckm" podStartSLOduration=3.451027017 podStartE2EDuration="11.246632616s" totalImagesPullingTime="7.795605599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:42.203913731 +0000 UTC m=+11.742254674" watchObservedRunningTime="2026-10-30 01:16:42.246632616 +0000 UTC m=+11.784973558" Oct 30 01:16:42.573503 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:42.573414 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:42.573817 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:42.573797 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:42.615810 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:42.615762 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-nvmrz" podStartSLOduration=3.639896319 podStartE2EDuration="11.61575088s" totalImagesPullingTime="7.975854561s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:42.247458822 +0000 UTC m=+11.785799767" watchObservedRunningTime="2026-10-30 01:16:42.61575088 +0000 UTC m=+12.154091821" Oct 30 01:16:42.902428 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:42.902405 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:42.902542 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:42.902497 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:42.902657 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:42.902643 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:42.902752 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:42.902728 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:43.125622 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:43.125557 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" event={"ID":"97b353f9-eeeb-4f6a-bdb7-204450f76167","Type":"ContainerStarted","Data":"b77ac6e7df2cf7d5c63b83e2a3f4fb39be6a141aa07516fa248fcfc62c2ba8cd"} Oct 30 01:16:43.129262 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:43.129236 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4slr" event={"ID":"335e4ad1-7154-4e2c-a170-d52aee2df045","Type":"ContainerStarted","Data":"19c076481d99dc37442b14f1ea8af7c393a52f2fd0d5540785459617513c58cd"} Oct 30 01:16:43.131711 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:43.131686 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-blx29" event={"ID":"f207f3a7-cf1c-4acc-8847-267e7c89e31f","Type":"ContainerStarted","Data":"12dc1e49f00cc99fd998bfaed52dff499ed204998620b7f5a6c04b764550021a"} Oct 30 01:16:43.214318 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:43.214273 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" podStartSLOduration=3.800255239 podStartE2EDuration="12.214259267s" totalImagesPullingTime="8.414004028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:43.171226364 +0000 UTC m=+12.709567305" watchObservedRunningTime="2026-10-30 01:16:43.214259267 +0000 UTC m=+12.752600259" Oct 30 01:16:43.214557 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:43.214533 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-blx29" podStartSLOduration=4.018138095 podStartE2EDuration="12.214524345s" totalImagesPullingTime="8.19638625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:43.21388524 +0000 UTC m=+12.752226178" watchObservedRunningTime="2026-10-30 01:16:43.214524345 +0000 UTC m=+12.752865288" Oct 30 01:16:43.900998 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:43.900811 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:43.901144 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:43.901111 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:44.135460 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:44.135376 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f4chz" event={"ID":"3db07759-c867-4de9-84c7-24d903510dca","Type":"ContainerStarted","Data":"516d47f7f696005d6020b3a4a56d8712bae3302fba9d1cf1c4423a70863a278a"} Oct 30 01:16:44.138021 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:44.137995 2578 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="19c076481d99dc37442b14f1ea8af7c393a52f2fd0d5540785459617513c58cd" exitCode=0 Oct 30 01:16:44.138141 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:44.138103 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4slr" event={"ID":"335e4ad1-7154-4e2c-a170-d52aee2df045","Type":"ContainerDied","Data":"19c076481d99dc37442b14f1ea8af7c393a52f2fd0d5540785459617513c58cd"} Oct 30 01:16:44.180978 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:44.180929 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f4chz" podStartSLOduration=2.660102304 podStartE2EDuration="13.180915788s" totalImagesPullingTime="10.520813484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:44.180241013 +0000 UTC m=+13.718581956" watchObservedRunningTime="2026-10-30 01:16:44.180915788 +0000 UTC m=+13.719256730" Oct 30 01:16:44.900638 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:44.900609 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:44.900790 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:44.900613 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:44.900790 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:44.900737 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:44.900790 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:44.900778 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:45.142731 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:45.142699 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4slr" event={"ID":"335e4ad1-7154-4e2c-a170-d52aee2df045","Type":"ContainerStarted","Data":"2aaccf018137369123e95024f6f7b108e421428647133dc1a47fa4a2040b5e2c"} Oct 30 01:16:45.901391 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:45.901360 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:45.901550 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:45.901505 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:46.147272 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:46.147236 2578 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="2aaccf018137369123e95024f6f7b108e421428647133dc1a47fa4a2040b5e2c" exitCode=0 Oct 30 01:16:46.147879 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:46.147298 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4slr" event={"ID":"335e4ad1-7154-4e2c-a170-d52aee2df045","Type":"ContainerDied","Data":"2aaccf018137369123e95024f6f7b108e421428647133dc1a47fa4a2040b5e2c"} Oct 30 01:16:46.901139 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:46.901106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:46.901317 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:46.901247 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:46.901643 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:46.901622 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:46.901738 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:46.901724 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:47.900916 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:47.900890 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:47.901413 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:47.900987 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:48.153802 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:48.153628 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" event={"ID":"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3","Type":"ContainerStarted","Data":"ea39e78845d820c46686395af5d3746f1e77935f7327f8bf91f2e084fc3bb887"} Oct 30 01:16:48.153888 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:48.153846 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" event={"ID":"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3","Type":"ContainerStarted","Data":"9bff0302cadd703154dc00d690924d05906b823758bd90b92184f513317eabcd"} Oct 30 01:16:48.662459 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:48.662248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:48.662597 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:48.662493 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:48.662597 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:48.662388 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:48.662597 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:48.662572 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret podName:8bc436c0-2b1b-477e-81aa-af1a1eff9779 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.662558323 +0000 UTC m=+34.200899243 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret") pod "global-pull-secret-syncer-rz57q" (UID: "8bc436c0-2b1b-477e-81aa-af1a1eff9779") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:48.662597 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:48.662572 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:48.662724 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:48.662608 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs podName:d8c54d89-eea4-45ee-80e5-fd0aba53e274 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.662598141 +0000 UTC m=+34.200939060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs") pod "network-metrics-daemon-p9qrg" (UID: "d8c54d89-eea4-45ee-80e5-fd0aba53e274") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:48.763394 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:48.763368 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:48.763502 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:48.763494 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:48.763534 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:48.763509 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:48.763534 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:48.763517 2578 projected.go:196] Error preparing data for projected volume kube-api-access-qq8g2 for pod openshift-network-diagnostics/network-check-target-4cnv8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:48.763593 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:48.763558 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2 podName:dbf09128-a8c1-4d32-958b-cbd01b892097 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.763547167 +0000 UTC m=+34.301888100 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qq8g2" (UniqueName: "kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2") pod "network-check-target-4cnv8" (UID: "dbf09128-a8c1-4d32-958b-cbd01b892097") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:48.900995 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:48.900966 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:48.900995 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:48.900997 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:48.901417 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:48.901095 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:48.901417 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:48.901156 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:49.158645 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:49.158612 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" event={"ID":"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3","Type":"ContainerStarted","Data":"f05d7982b2f80e1e46a18cad0e7f1c1d732ff1849006e9e24af3a9e825dd1c47"} Oct 30 01:16:49.158785 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:49.158702 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" event={"ID":"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3","Type":"ContainerStarted","Data":"3e518cab0bb18c353aa26fd6dae49b619685431b65b47ebaf8f75b78e14e6218"} Oct 30 01:16:49.158785 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:49.158755 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" event={"ID":"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3","Type":"ContainerStarted","Data":"288fbfe2ac742091f6a3e4f81a8c563e41b9d028c65a3ff86f1331a6601d7472"} Oct 30 01:16:49.158869 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:49.158801 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" event={"ID":"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3","Type":"ContainerStarted","Data":"7b2d6f2af0a4c7447427b51f5e9ed33de1237c061b8a2db4a899caa1efeb970f"} Oct 30 01:16:49.161402 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:49.161368 2578 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="6bdd3ffb117ec1d05af86da3ccae99e968ac457bfe61bf2532baba3ba3f80279" exitCode=0 Oct 30 01:16:49.161520 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:49.161412 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4slr" event={"ID":"335e4ad1-7154-4e2c-a170-d52aee2df045","Type":"ContainerDied","Data":"6bdd3ffb117ec1d05af86da3ccae99e968ac457bfe61bf2532baba3ba3f80279"} Oct 30 01:16:49.772476 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:49.772449 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:49.772617 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:49.772579 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:16:49.773017 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:49.773003 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:16:49.901168 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:49.901143 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:49.901566 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:49.901239 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:50.169256 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:50.169184 2578 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="bb140a901cfd5d99ac10a3734ac47d8976627c8768d3a98d3e7cb6317ce5097c" exitCode=0 Oct 30 01:16:50.169384 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:50.169264 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4slr" event={"ID":"335e4ad1-7154-4e2c-a170-d52aee2df045","Type":"ContainerDied","Data":"bb140a901cfd5d99ac10a3734ac47d8976627c8768d3a98d3e7cb6317ce5097c"} Oct 30 01:16:50.901320 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:50.901229 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:50.901792 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:50.901388 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:50.901792 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:50.901424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:50.901792 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:50.901510 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:51.174708 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:51.174633 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" event={"ID":"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3","Type":"ContainerStarted","Data":"3cb25ef4f21c82e5f63fd67df758c1742bd0c3c6c76f36cfe7d54dd38b11685a"} Oct 30 01:16:51.177899 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:51.177873 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4slr" event={"ID":"335e4ad1-7154-4e2c-a170-d52aee2df045","Type":"ContainerStarted","Data":"5dc256ac13ea6c3f49fe4646433109bbb1e2eb9930ec1fd1eaaa60f76366d2d1"} Oct 30 01:16:51.260438 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:51.260390 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v4slr" podStartSLOduration=9.291973847 podStartE2EDuration="20.260377209s" totalImagesPullingTime="10.968403362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:51.260116688 +0000 UTC m=+20.798457630" watchObservedRunningTime="2026-10-30 01:16:51.260377209 +0000 UTC m=+20.798718151" Oct 30 01:16:51.901343 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:51.901300 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:51.901787 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:51.901399 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:52.901025 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:52.900933 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:52.901025 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:52.900964 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:52.901235 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:52.901046 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:52.901235 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:52.901156 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:53.186099 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:53.186015 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" event={"ID":"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3","Type":"ContainerStarted","Data":"0f89a3697261190926bcfb4951d7af40b98fc34b656e18eb12890058d05d13ea"} Oct 30 01:16:53.186793 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:53.186456 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:53.186793 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:53.186528 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:53.201444 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:53.201420 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:53.201573 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:53.201535 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:16:53.281816 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:53.281768 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" podStartSLOduration=7.519124283 podStartE2EDuration="22.281755873s" totalImagesPullingTime="14.76263159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:53.28156426 +0000 UTC m=+22.819905203" watchObservedRunningTime="2026-10-30 01:16:53.281755873 +0000 UTC m=+22.820096818" Oct 30 01:16:53.901200 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:53.901171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:53.901432 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:53.901281 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:54.187727 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:54.187661 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:16:54.638682 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:54.638478 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rz57q"] Oct 30 01:16:54.638821 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:54.638781 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:54.638922 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:54.638897 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:54.639061 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:54.639044 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4cnv8"] Oct 30 01:16:54.639141 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:54.639129 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:54.639224 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:54.639208 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:54.652716 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:54.652694 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p9qrg"] Oct 30 01:16:54.652815 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:54.652763 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:54.652862 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:54.652828 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:55.190374 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:55.190341 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:16:55.900928 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:55.900894 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:55.900928 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:55.900925 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:55.901121 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:55.900940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:55.901121 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:55.901048 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:55.901187 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:55.901166 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:55.901265 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:55.901248 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:57.901255 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:57.901227 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:57.901745 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:57.901320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:57.901745 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:57.901358 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:57.901745 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:57.901392 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:57.901745 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:57.901409 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:57.901745 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:57.901470 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:16:59.900787 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:59.900752 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:16:59.901280 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:59.900770 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:16:59.901280 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:59.900886 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4cnv8" podUID="dbf09128-a8c1-4d32-958b-cbd01b892097" Oct 30 01:16:59.901280 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:16:59.901013 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:16:59.901280 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:59.901004 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rz57q" podUID="8bc436c0-2b1b-477e-81aa-af1a1eff9779" Oct 30 01:16:59.901280 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:16:59.901111 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p9qrg" podUID="d8c54d89-eea4-45ee-80e5-fd0aba53e274" Oct 30 01:17:00.718459 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.718436 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeReady" Oct 30 01:17:00.718567 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.718520 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 01:17:00.770658 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.770638 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tw27b"] Oct 30 01:17:00.773620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.773603 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.775568 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.775535 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-54db9d5677-ptwtb"] Oct 30 01:17:00.778476 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.778460 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c9s9t"] Oct 30 01:17:00.778572 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.778556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.781289 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.781269 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-655b484568-p27mb"] Oct 30 01:17:00.781401 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.781384 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c9s9t" Oct 30 01:17:00.784082 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.784064 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xsnwp"] Oct 30 01:17:00.784234 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.784187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.787637 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.787480 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-pb877"] Oct 30 01:17:00.787637 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.787612 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xsnwp" Oct 30 01:17:00.790314 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.790295 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kzdph"] Oct 30 01:17:00.790413 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.790387 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:00.793020 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.793004 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-67bf8c5c5-mtkk5"] Oct 30 01:17:00.793120 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.793106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:00.795847 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.795828 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf"] Oct 30 01:17:00.795917 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.795859 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.796260 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.796098 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.796260 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.796218 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 01:17:00.798539 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.798521 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 01:17:00.798539 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.798532 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4q66s"] Oct 30 01:17:00.798652 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.798546 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tgxzm\"" Oct 30 01:17:00.798652 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.798570 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.798714 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.798667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:00.801683 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.801666 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-64j9l"] Oct 30 01:17:00.801786 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.801769 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4q66s" Oct 30 01:17:00.805016 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.805001 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb"] Oct 30 01:17:00.805120 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.805106 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:00.812010 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.811961 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r"] Oct 30 01:17:00.812281 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.812262 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:00.814970 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.814952 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv"] Oct 30 01:17:00.815565 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.815544 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r" Oct 30 01:17:00.816028 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.816012 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 01:17:00.818773 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.818741 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v"] Oct 30 01:17:00.819075 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.819057 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv" Oct 30 01:17:00.821449 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.821430 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qdpcp"] Oct 30 01:17:00.821702 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.821680 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:00.825751 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.825705 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lh2vb"] Oct 30 01:17:00.825863 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.825820 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:00.832134 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.832099 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tw27b"] Oct 30 01:17:00.834072 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.834044 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54db9d5677-ptwtb"] Oct 30 01:17:00.834516 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.834495 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xsnwp"] Oct 30 01:17:00.834516 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.834534 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-655b484568-p27mb"] Oct 30 01:17:00.834691 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.834556 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-pb877"] Oct 30 01:17:00.834691 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.834576 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kzdph"] Oct 30 01:17:00.834691 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.834597 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r"] Oct 30 01:17:00.834691 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.834616 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67bf8c5c5-mtkk5"] Oct 30 01:17:00.835113 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.834718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:00.835163 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.835112 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv"] Oct 30 01:17:00.838104 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.838084 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf"] Oct 30 01:17:00.838178 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.838130 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qdpcp"] Oct 30 01:17:00.839022 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.839005 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c9s9t"] Oct 30 01:17:00.852625 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.852603 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v"] Oct 30 01:17:00.856690 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.856673 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-trusted-ca\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.856763 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.856698 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-installation-pull-secrets\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.856763 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.856719 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-serving-cert\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.856763 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.856735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-image-registry-private-configuration\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.856763 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.856752 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:00.856930 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.856818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k4bh\" (UniqueName: \"kubernetes.io/projected/b4e79600-74c8-4e22-8fce-fbd02aca1e38-kube-api-access-6k4bh\") pod \"kas-connection-checker-685895c7c9-xsnwp\" (UID: \"b4e79600-74c8-4e22-8fce-fbd02aca1e38\") " pod="kube-system/kas-connection-checker-685895c7c9-xsnwp" Oct 30 01:17:00.856930 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.856881 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:00.857008 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.856933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwz2\" (UniqueName: \"kubernetes.io/projected/c9b752e3-e38a-4d0f-babd-7922d6a2e872-kube-api-access-gnwz2\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:00.857008 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.856979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/712bffb7-97f2-422a-ba7b-cce9eafccba4-serving-cert\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:00.857104 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwp4f\" (UniqueName: \"kubernetes.io/projected/4a8853b4-1b14-4b16-81d8-007e7827d8dd-kube-api-access-lwp4f\") pod \"kas-connection-checker-685895c7c9-c9s9t\" (UID: \"4a8853b4-1b14-4b16-81d8-007e7827d8dd\") " pod="kube-system/kas-connection-checker-685895c7c9-c9s9t" Oct 30 01:17:00.857104 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7082f68a-2079-426e-997f-1d3ba3997ba6-ca-trust-extracted\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.857176 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jblp7\" (UniqueName: \"kubernetes.io/projected/a310cb37-d20b-4b25-9796-c5f56c44adac-kube-api-access-jblp7\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.857176 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857142 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-bound-sa-token\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.857176 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-certificates\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.857393 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqtw\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-kube-api-access-2tqtw\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.857393 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857339 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.857496 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857403 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.857496 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6khjk\" (UniqueName: \"kubernetes.io/projected/712bffb7-97f2-422a-ba7b-cce9eafccba4-kube-api-access-6khjk\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:00.857565 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857510 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-trusted-ca\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.857605 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/712bffb7-97f2-422a-ba7b-cce9eafccba4-config\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:00.857642 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857607 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46q9r\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-kube-api-access-46q9r\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.857683 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857656 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/806fd3b6-00de-4f42-8c35-e638d3835ec7-ca-trust-extracted\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.857683 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857671 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-stats-auth\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.857764 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857742 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-default-certificate\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.857808 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.857808 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9b752e3-e38a-4d0f-babd-7922d6a2e872-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:00.857901 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857820 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.857901 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-installation-pull-secrets\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.857901 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:00.858076 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-config\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.858076 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857928 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-image-registry-private-configuration\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.858076 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.857957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-trusted-ca\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.858076 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.858019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-bound-sa-token\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.858076 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.858043 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-certificates\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.858076 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.858065 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8q47\" (UniqueName: \"kubernetes.io/projected/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-kube-api-access-f8q47\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.860237 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.860220 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb"] Oct 30 01:17:00.865882 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.865864 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lh2vb"] Oct 30 01:17:00.867231 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.867211 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4q66s"] Oct 30 01:17:00.868167 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.868147 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-64j9l"] Oct 30 01:17:00.897087 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897060 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.897183 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897106 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 01:17:00.897358 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897342 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 01:17:00.897436 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897425 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 01:17:00.897500 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897437 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9gpf8\"" Oct 30 01:17:00.897500 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897448 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 01:17:00.897500 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897459 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:17:00.897500 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897495 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:17:00.897672 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897541 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 01:17:00.897880 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897860 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 01:17:00.898169 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.897971 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 01:17:00.898169 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.898022 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.898169 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.898033 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4mfm\"" Oct 30 01:17:00.898169 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.898039 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.898169 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.898047 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 01:17:00.898169 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.898099 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.898169 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.898101 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:17:00.898169 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.898101 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.898535 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.898195 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.898535 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.898480 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.898930 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.898913 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.899239 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.899121 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 01:17:00.900745 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.900436 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 01:17:00.900745 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.900699 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 01:17:00.901115 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.900861 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.901115 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.900981 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.901115 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.901088 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 01:17:00.901240 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.901140 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 01:17:00.901489 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.901461 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.902925 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.902908 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xd5g5\"" Oct 30 01:17:00.903212 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.903191 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 01:17:00.903524 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.903507 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.903801 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.903784 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 01:17:00.903901 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.903875 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zbwdc\"" Oct 30 01:17:00.904131 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.904100 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.904281 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.904263 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.904378 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.904361 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 01:17:00.904514 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.904496 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.904619 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.904536 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 01:17:00.904619 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.904548 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.905043 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905023 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mf6zs\"" Oct 30 01:17:00.905119 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905024 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:00.905545 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905525 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.905610 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905552 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lvp69\"" Oct 30 01:17:00.905659 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905614 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms58v\"" Oct 30 01:17:00.905659 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905653 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.905749 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905677 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-f5n9z\"" Oct 30 01:17:00.905749 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905708 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 01:17:00.905749 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905736 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:00.905873 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905799 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qgx7d\"" Oct 30 01:17:00.905873 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.905850 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cwfw8\"" Oct 30 01:17:00.906039 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.906020 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 01:17:00.906145 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.906075 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zs626\"" Oct 30 01:17:00.906145 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.906125 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-58v5x\"" Oct 30 01:17:00.913175 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.913160 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 01:17:00.913264 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.913249 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 01:17:00.958396 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958359 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.958515 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958405 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khjk\" (UniqueName: \"kubernetes.io/projected/712bffb7-97f2-422a-ba7b-cce9eafccba4-kube-api-access-6khjk\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:00.958515 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-trusted-ca\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.958515 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958455 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:00.958515 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/712bffb7-97f2-422a-ba7b-cce9eafccba4-config\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:00.958515 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.958506 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 01:17:00.958761 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-config-volume\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:00.958761 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.958583 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:01.458564172 +0000 UTC m=+30.996905109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : secret "router-metrics-certs-default" not found Oct 30 01:17:00.958761 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:00.958761 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46q9r\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-kube-api-access-46q9r\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.958761 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958743 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx866\" (UniqueName: \"kubernetes.io/projected/495d54bf-6c21-46ab-badd-8ece131d6618-kube-api-access-sx866\") pod \"network-check-source-6f78d59f99-rgvjv\" (UID: \"495d54bf-6c21-46ab-badd-8ece131d6618\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv" Oct 30 01:17:00.959014 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958811 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-snapshots\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:00.959014 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958845 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2cf9\" (UniqueName: \"kubernetes.io/projected/8855c2e7-9182-4ae6-b96a-50baa69d9a67-kube-api-access-z2cf9\") pod \"ingress-canary-lh2vb\" (UID: \"8855c2e7-9182-4ae6-b96a-50baa69d9a67\") " pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:00.959014 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzmb8\" (UniqueName: \"kubernetes.io/projected/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-kube-api-access-bzmb8\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:00.959014 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nzwm\" (UniqueName: \"kubernetes.io/projected/083c7373-177e-46cb-99c1-f9509121668a-kube-api-access-8nzwm\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:00.959014 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/806fd3b6-00de-4f42-8c35-e638d3835ec7-ca-trust-extracted\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.959014 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-stats-auth\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.959014 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.958996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-tmp-dir\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:00.959364 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-default-certificate\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.959364 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959054 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.959364 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959079 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9b752e3-e38a-4d0f-babd-7922d6a2e872-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:00.959364 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959094 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/712bffb7-97f2-422a-ba7b-cce9eafccba4-config\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:00.959364 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959102 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.959364 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb940654-4320-4340-a2d5-9450bc68f40f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:00.959364 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jwhg\" (UniqueName: \"kubernetes.io/projected/bb940654-4320-4340-a2d5-9450bc68f40f-kube-api-access-6jwhg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:00.959364 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959321 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/806fd3b6-00de-4f42-8c35-e638d3835ec7-ca-trust-extracted\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.959753 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959382 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-trusted-ca\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.959753 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.959499 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:17:00.959753 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.959513 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54db9d5677-ptwtb: secret "image-registry-tls" not found Oct 30 01:17:00.959753 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.959599 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:01.459584369 +0000 UTC m=+30.997925314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : configmap references non-existent config key: service-ca.crt Oct 30 01:17:00.959753 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.959666 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls podName:806fd3b6-00de-4f42-8c35-e638d3835ec7 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:01.459649034 +0000 UTC m=+30.997989954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls") pod "image-registry-54db9d5677-ptwtb" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7") : secret "image-registry-tls" not found Oct 30 01:17:00.959753 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-installation-pull-secrets\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.959753 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959732 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-config\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959797 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb940654-4320-4340-a2d5-9450bc68f40f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959825 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9b752e3-e38a-4d0f-babd-7922d6a2e872-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959841 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-image-registry-private-configuration\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-trusted-ca\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8855c2e7-9182-4ae6-b96a-50baa69d9a67-cert\") pod \"ingress-canary-lh2vb\" (UID: \"8855c2e7-9182-4ae6-b96a-50baa69d9a67\") " pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-bound-sa-token\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959944 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-certificates\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.959970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8q47\" (UniqueName: \"kubernetes.io/projected/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-kube-api-access-f8q47\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-trusted-ca\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-installation-pull-secrets\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-serving-cert\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.960102 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/083c7373-177e-46cb-99c1-f9509121668a-serving-cert\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:00.960746 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pzfs\" (UniqueName: \"kubernetes.io/projected/1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5-kube-api-access-8pzfs\") pod \"kas-connection-checker-685895c7c9-4q66s\" (UID: \"1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5\") " pod="kube-system/kas-connection-checker-685895c7c9-4q66s" Oct 30 01:17:00.960746 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jb47\" (UniqueName: \"kubernetes.io/projected/4588e6fe-4b1e-4e62-8528-7411e598b444-kube-api-access-9jb47\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:00.960746 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960203 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-image-registry-private-configuration\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.960746 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960231 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:00.960746 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4bh\" (UniqueName: \"kubernetes.io/projected/b4e79600-74c8-4e22-8fce-fbd02aca1e38-kube-api-access-6k4bh\") pod \"kas-connection-checker-685895c7c9-xsnwp\" (UID: \"b4e79600-74c8-4e22-8fce-fbd02aca1e38\") " pod="kube-system/kas-connection-checker-685895c7c9-xsnwp" Oct 30 01:17:00.960746 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:00.960746 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-service-ca-bundle\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:00.960746 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960356 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcv6t\" (UniqueName: \"kubernetes.io/projected/98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db-kube-api-access-vcv6t\") pod \"volume-data-source-validator-58c9f5f46d-z5d5r\" (UID: \"98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r" Oct 30 01:17:00.960746 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.960357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:00.961191 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.961102 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-config\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.961508 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.961490 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:17:00.961698 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.961686 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert podName:0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:01.461668516 +0000 UTC m=+31.000009449 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kzdph" (UID: "0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41") : secret "networking-console-plugin-cert" not found Oct 30 01:17:00.963912 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.963886 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-stats-auth\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.963912 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.963897 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-image-registry-private-configuration\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.963912 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.963904 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-installation-pull-secrets\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.964092 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.963901 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-default-certificate\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.964092 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.963916 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-installation-pull-secrets\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.964092 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.963964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:00.964092 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.963996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwz2\" (UniqueName: \"kubernetes.io/projected/c9b752e3-e38a-4d0f-babd-7922d6a2e872-kube-api-access-gnwz2\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:00.964092 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/712bffb7-97f2-422a-ba7b-cce9eafccba4-serving-cert\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:00.964092 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.964042 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:00.964425 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.964108 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls podName:c9b752e3-e38a-4d0f-babd-7922d6a2e872 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:01.464094961 +0000 UTC m=+31.002435893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pb877" (UID: "c9b752e3-e38a-4d0f-babd-7922d6a2e872") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:00.964425 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964137 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-trusted-ca\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.964425 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964273 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwp4f\" (UniqueName: \"kubernetes.io/projected/4a8853b4-1b14-4b16-81d8-007e7827d8dd-kube-api-access-lwp4f\") pod \"kas-connection-checker-685895c7c9-c9s9t\" (UID: \"4a8853b4-1b14-4b16-81d8-007e7827d8dd\") " pod="kube-system/kas-connection-checker-685895c7c9-c9s9t" Oct 30 01:17:00.964425 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7082f68a-2079-426e-997f-1d3ba3997ba6-ca-trust-extracted\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.964425 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964368 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-certificates\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.964834 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964536 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jblp7\" (UniqueName: \"kubernetes.io/projected/a310cb37-d20b-4b25-9796-c5f56c44adac-kube-api-access-jblp7\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.964834 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964644 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-bound-sa-token\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.964834 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964705 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7082f68a-2079-426e-997f-1d3ba3997ba6-ca-trust-extracted\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.964834 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964761 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-certificates\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.965048 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964885 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-tmp\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:00.965048 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964940 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqtw\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-kube-api-access-2tqtw\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.965048 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964965 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:00.965048 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.964993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.965246 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.965090 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-certificates\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.965246 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.965123 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:17:00.965246 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.965137 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655b484568-p27mb: secret "image-registry-tls" not found Oct 30 01:17:00.965246 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:00.965187 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls podName:7082f68a-2079-426e-997f-1d3ba3997ba6 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:01.465172229 +0000 UTC m=+31.003513152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls") pod "image-registry-655b484568-p27mb" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6") : secret "image-registry-tls" not found Oct 30 01:17:00.965657 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.965636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-trusted-ca\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.966020 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.966003 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-serving-cert\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.966258 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.966244 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/712bffb7-97f2-422a-ba7b-cce9eafccba4-serving-cert\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:00.966672 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.966656 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-image-registry-private-configuration\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.979710 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.979691 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khjk\" (UniqueName: \"kubernetes.io/projected/712bffb7-97f2-422a-ba7b-cce9eafccba4-kube-api-access-6khjk\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:00.983372 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.983350 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46q9r\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-kube-api-access-46q9r\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.985521 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.985502 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-bound-sa-token\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.988432 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.988408 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4bh\" (UniqueName: \"kubernetes.io/projected/b4e79600-74c8-4e22-8fce-fbd02aca1e38-kube-api-access-6k4bh\") pod \"kas-connection-checker-685895c7c9-xsnwp\" (UID: \"b4e79600-74c8-4e22-8fce-fbd02aca1e38\") " pod="kube-system/kas-connection-checker-685895c7c9-xsnwp" Oct 30 01:17:00.990730 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.990705 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-bound-sa-token\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:00.990824 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.990789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwz2\" (UniqueName: \"kubernetes.io/projected/c9b752e3-e38a-4d0f-babd-7922d6a2e872-kube-api-access-gnwz2\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:00.991085 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.991065 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8q47\" (UniqueName: \"kubernetes.io/projected/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-kube-api-access-f8q47\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:00.992409 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.992384 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jblp7\" (UniqueName: \"kubernetes.io/projected/a310cb37-d20b-4b25-9796-c5f56c44adac-kube-api-access-jblp7\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:00.993190 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.993169 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqtw\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-kube-api-access-2tqtw\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:00.993673 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:00.993657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwp4f\" (UniqueName: \"kubernetes.io/projected/4a8853b4-1b14-4b16-81d8-007e7827d8dd-kube-api-access-lwp4f\") pod \"kas-connection-checker-685895c7c9-c9s9t\" (UID: \"4a8853b4-1b14-4b16-81d8-007e7827d8dd\") " pod="kube-system/kas-connection-checker-685895c7c9-c9s9t" Oct 30 01:17:01.065669 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.065636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:01.065846 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.065709 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:01.065846 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.065735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-config-volume\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:01.065846 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.065764 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:01.065846 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.065790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx866\" (UniqueName: \"kubernetes.io/projected/495d54bf-6c21-46ab-badd-8ece131d6618-kube-api-access-sx866\") pod \"network-check-source-6f78d59f99-rgvjv\" (UID: \"495d54bf-6c21-46ab-badd-8ece131d6618\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv" Oct 30 01:17:01.065846 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.065804 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 01:17:01.065846 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.065819 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-snapshots\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.065846 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.065834 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 01:17:01.065846 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.065846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2cf9\" (UniqueName: \"kubernetes.io/projected/8855c2e7-9182-4ae6-b96a-50baa69d9a67-kube-api-access-z2cf9\") pod \"ingress-canary-lh2vb\" (UID: \"8855c2e7-9182-4ae6-b96a-50baa69d9a67\") " pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.065875 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls podName:4588e6fe-4b1e-4e62-8528-7411e598b444 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:01.565855947 +0000 UTC m=+31.104196885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54ddb" (UID: "4588e6fe-4b1e-4e62-8528-7411e598b444") : secret "samples-operator-tls" not found Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.065910 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls podName:1e24a38d-9f35-40bf-bb90-57fee63d8ac0 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:01.565893673 +0000 UTC m=+31.104234607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls") pod "dns-default-qdpcp" (UID: "1e24a38d-9f35-40bf-bb90-57fee63d8ac0") : secret "dns-default-metrics-tls" not found Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.065929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmb8\" (UniqueName: \"kubernetes.io/projected/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-kube-api-access-bzmb8\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.065967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzwm\" (UniqueName: \"kubernetes.io/projected/083c7373-177e-46cb-99c1-f9509121668a-kube-api-access-8nzwm\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066000 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-tmp-dir\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb940654-4320-4340-a2d5-9450bc68f40f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jwhg\" (UniqueName: \"kubernetes.io/projected/bb940654-4320-4340-a2d5-9450bc68f40f-kube-api-access-6jwhg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb940654-4320-4340-a2d5-9450bc68f40f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066128 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8855c2e7-9182-4ae6-b96a-50baa69d9a67-cert\") pod \"ingress-canary-lh2vb\" (UID: \"8855c2e7-9182-4ae6-b96a-50baa69d9a67\") " pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/083c7373-177e-46cb-99c1-f9509121668a-serving-cert\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pzfs\" (UniqueName: \"kubernetes.io/projected/1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5-kube-api-access-8pzfs\") pod \"kas-connection-checker-685895c7c9-4q66s\" (UID: \"1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5\") " pod="kube-system/kas-connection-checker-685895c7c9-4q66s" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066221 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jb47\" (UniqueName: \"kubernetes.io/projected/4588e6fe-4b1e-4e62-8528-7411e598b444-kube-api-access-9jb47\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066259 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.066375 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-service-ca-bundle\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.067120 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066347 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcv6t\" (UniqueName: \"kubernetes.io/projected/98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db-kube-api-access-vcv6t\") pod \"volume-data-source-validator-58c9f5f46d-z5d5r\" (UID: \"98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r" Oct 30 01:17:01.067120 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066441 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-tmp\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.067120 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066668 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-config-volume\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:01.067120 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066696 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-snapshots\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.067120 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066726 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-tmp\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.067120 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.066770 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:01.067120 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.067002 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-tmp-dir\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:01.067480 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.067461 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb940654-4320-4340-a2d5-9450bc68f40f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:01.068187 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.068159 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-service-ca-bundle\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.068468 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.068447 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.069592 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.069569 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8855c2e7-9182-4ae6-b96a-50baa69d9a67-cert\") pod \"ingress-canary-lh2vb\" (UID: \"8855c2e7-9182-4ae6-b96a-50baa69d9a67\") " pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:01.069592 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.069574 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb940654-4320-4340-a2d5-9450bc68f40f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:01.069706 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.069614 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/083c7373-177e-46cb-99c1-f9509121668a-serving-cert\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.086783 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.086730 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:01.089238 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.089213 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jwhg\" (UniqueName: \"kubernetes.io/projected/bb940654-4320-4340-a2d5-9450bc68f40f-kube-api-access-6jwhg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:01.089531 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.089476 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx866\" (UniqueName: \"kubernetes.io/projected/495d54bf-6c21-46ab-badd-8ece131d6618-kube-api-access-sx866\") pod \"network-check-source-6f78d59f99-rgvjv\" (UID: \"495d54bf-6c21-46ab-badd-8ece131d6618\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv" Oct 30 01:17:01.090914 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.090883 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2cf9\" (UniqueName: \"kubernetes.io/projected/8855c2e7-9182-4ae6-b96a-50baa69d9a67-kube-api-access-z2cf9\") pod \"ingress-canary-lh2vb\" (UID: \"8855c2e7-9182-4ae6-b96a-50baa69d9a67\") " pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:01.091976 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.091951 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pzfs\" (UniqueName: \"kubernetes.io/projected/1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5-kube-api-access-8pzfs\") pod \"kas-connection-checker-685895c7c9-4q66s\" (UID: \"1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5\") " pod="kube-system/kas-connection-checker-685895c7c9-4q66s" Oct 30 01:17:01.092624 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.092483 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzwm\" (UniqueName: \"kubernetes.io/projected/083c7373-177e-46cb-99c1-f9509121668a-kube-api-access-8nzwm\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.093355 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.093295 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmb8\" (UniqueName: \"kubernetes.io/projected/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-kube-api-access-bzmb8\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:01.094053 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.094028 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcv6t\" (UniqueName: \"kubernetes.io/projected/98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db-kube-api-access-vcv6t\") pod \"volume-data-source-validator-58c9f5f46d-z5d5r\" (UID: \"98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r" Oct 30 01:17:01.095775 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.095753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jb47\" (UniqueName: \"kubernetes.io/projected/4588e6fe-4b1e-4e62-8528-7411e598b444-kube-api-access-9jb47\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:01.101856 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.101836 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-c9s9t" Oct 30 01:17:01.121403 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.121373 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xsnwp" Oct 30 01:17:01.164812 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.164578 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:01.170844 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.170786 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-4q66s" Oct 30 01:17:01.178216 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.178155 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:01.189774 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.189712 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r" Oct 30 01:17:01.203035 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.198093 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv" Oct 30 01:17:01.204370 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.203607 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:01.209566 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.209475 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:01.422847 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.422824 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-c9s9t"] Oct 30 01:17:01.423457 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:01.423389 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8853b4_1b14_4b16_81d8_007e7827d8dd.slice/crio-26a4ee28be21d0a49084db02816636f8b892cb1441c9cd007faaedbb29923b4a WatchSource:0}: Error finding container 26a4ee28be21d0a49084db02816636f8b892cb1441c9cd007faaedbb29923b4a: Status 404 returned error can't find the container with id 26a4ee28be21d0a49084db02816636f8b892cb1441c9cd007faaedbb29923b4a Oct 30 01:17:01.432499 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.432472 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-tw27b"] Oct 30 01:17:01.434943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:01.434918 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cb0d17_b098_408e_92b6_b5e81a8bb9cf.slice/crio-e22fe9cb499d4fedab46c2f49f47021cc3c5614e5e0be3322b6117465b462b08 WatchSource:0}: Error finding container e22fe9cb499d4fedab46c2f49f47021cc3c5614e5e0be3322b6117465b462b08: Status 404 returned error can't find the container with id e22fe9cb499d4fedab46c2f49f47021cc3c5614e5e0be3322b6117465b462b08 Oct 30 01:17:01.464312 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.464287 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xsnwp"] Oct 30 01:17:01.464943 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:01.464915 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e79600_74c8_4e22_8fce_fbd02aca1e38.slice/crio-dd4f7bdb4814f7c89507166194d8182512d9eae411e8af641cae73ec672b1274 WatchSource:0}: Error finding container dd4f7bdb4814f7c89507166194d8182512d9eae411e8af641cae73ec672b1274: Status 404 returned error can't find the container with id dd4f7bdb4814f7c89507166194d8182512d9eae411e8af641cae73ec672b1274 Oct 30 01:17:01.472190 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.472171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:01.472271 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.472200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:01.472318 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472305 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:01.472318 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472308 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:17:01.472429 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.472371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:01.472429 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472426 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:17:01.472486 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472374 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls podName:c9b752e3-e38a-4d0f-babd-7922d6a2e872 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.47236025 +0000 UTC m=+32.010701169 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pb877" (UID: "c9b752e3-e38a-4d0f-babd-7922d6a2e872") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:01.472486 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472436 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655b484568-p27mb: secret "image-registry-tls" not found Oct 30 01:17:01.472486 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472459 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert podName:0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.472445778 +0000 UTC m=+32.010786698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kzdph" (UID: "0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41") : secret "networking-console-plugin-cert" not found Oct 30 01:17:01.472486 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472482 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls podName:7082f68a-2079-426e-997f-1d3ba3997ba6 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.472476161 +0000 UTC m=+32.010817081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls") pod "image-registry-655b484568-p27mb" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6") : secret "image-registry-tls" not found Oct 30 01:17:01.472612 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.472504 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:01.472612 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.472555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:01.472666 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472608 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 01:17:01.472666 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472643 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.472630217 +0000 UTC m=+32.010971138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : secret "router-metrics-certs-default" not found Oct 30 01:17:01.472727 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472673 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:17:01.472727 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472681 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54db9d5677-ptwtb: secret "image-registry-tls" not found Oct 30 01:17:01.472727 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472704 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls podName:806fd3b6-00de-4f42-8c35-e638d3835ec7 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.472697261 +0000 UTC m=+32.011038180 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls") pod "image-registry-54db9d5677-ptwtb" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7") : secret "image-registry-tls" not found Oct 30 01:17:01.472812 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.472728 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:01.472847 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.472815 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.472808089 +0000 UTC m=+32.011149008 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : configmap references non-existent config key: service-ca.crt Oct 30 01:17:01.537016 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:01.536972 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495d54bf_6c21_46ab_badd_8ece131d6618.slice/crio-b3613ef44430a444c8560394f44d0ef7210b83d832738f1907fc15b6d90c131b WatchSource:0}: Error finding container b3613ef44430a444c8560394f44d0ef7210b83d832738f1907fc15b6d90c131b: Status 404 returned error can't find the container with id b3613ef44430a444c8560394f44d0ef7210b83d832738f1907fc15b6d90c131b Oct 30 01:17:01.538542 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.537694 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv"] Oct 30 01:17:01.539274 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.539253 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v"] Oct 30 01:17:01.540255 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.540159 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf"] Oct 30 01:17:01.543206 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:01.543180 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712bffb7_97f2_422a_ba7b_cce9eafccba4.slice/crio-17cb1ef5e6ccb32fd7a4049f9d2901ee7acde9ff5e2b8100424aec55d7637def WatchSource:0}: Error finding container 17cb1ef5e6ccb32fd7a4049f9d2901ee7acde9ff5e2b8100424aec55d7637def: Status 404 returned error can't find the container with id 17cb1ef5e6ccb32fd7a4049f9d2901ee7acde9ff5e2b8100424aec55d7637def Oct 30 01:17:01.562020 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.561969 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-4q66s"] Oct 30 01:17:01.565684 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:01.565655 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dcbad1e_b7e0_4dac_a2bc_55ddf6bb55f5.slice/crio-b9c6b81d1b651e648a6f8dbe14cccc2c4e2172dd5f3ecc71e9a0deb4fc2a72a8 WatchSource:0}: Error finding container b9c6b81d1b651e648a6f8dbe14cccc2c4e2172dd5f3ecc71e9a0deb4fc2a72a8: Status 404 returned error can't find the container with id b9c6b81d1b651e648a6f8dbe14cccc2c4e2172dd5f3ecc71e9a0deb4fc2a72a8 Oct 30 01:17:01.573130 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.573106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:01.573213 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.573175 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:01.573258 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.573239 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 01:17:01.573314 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.573304 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls podName:4588e6fe-4b1e-4e62-8528-7411e598b444 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.573286472 +0000 UTC m=+32.111627396 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54ddb" (UID: "4588e6fe-4b1e-4e62-8528-7411e598b444") : secret "samples-operator-tls" not found Oct 30 01:17:01.573511 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.573363 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 01:17:01.573511 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:01.573420 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls podName:1e24a38d-9f35-40bf-bb90-57fee63d8ac0 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.573403381 +0000 UTC m=+32.111744318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls") pod "dns-default-qdpcp" (UID: "1e24a38d-9f35-40bf-bb90-57fee63d8ac0") : secret "dns-default-metrics-tls" not found Oct 30 01:17:01.600967 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.600944 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lh2vb"] Oct 30 01:17:01.601649 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:01.601621 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8855c2e7_9182_4ae6_b96a_50baa69d9a67.slice/crio-49a14e50a6ab971df021c9e1e711e0da45528de7acef663ee026eadb4d2d11a8 WatchSource:0}: Error finding container 49a14e50a6ab971df021c9e1e711e0da45528de7acef663ee026eadb4d2d11a8: Status 404 returned error can't find the container with id 49a14e50a6ab971df021c9e1e711e0da45528de7acef663ee026eadb4d2d11a8 Oct 30 01:17:01.616099 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.616076 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-64j9l"] Oct 30 01:17:01.619900 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:01.619868 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083c7373_177e_46cb_99c1_f9509121668a.slice/crio-5586bcc51fe3d1f925e6091add497847856a02c5959decadb4c2a0144982ba6e WatchSource:0}: Error finding container 5586bcc51fe3d1f925e6091add497847856a02c5959decadb4c2a0144982ba6e: Status 404 returned error can't find the container with id 5586bcc51fe3d1f925e6091add497847856a02c5959decadb4c2a0144982ba6e Oct 30 01:17:01.646217 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.646195 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r"] Oct 30 01:17:01.646470 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:01.646447 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98bbc2c8_1ab4_4ebc_aea0_fffcaf91d4db.slice/crio-659e8018552c405d80f18629929fddbb026361e9a951ccf95a172f93321cd319 WatchSource:0}: Error finding container 659e8018552c405d80f18629929fddbb026361e9a951ccf95a172f93321cd319: Status 404 returned error can't find the container with id 659e8018552c405d80f18629929fddbb026361e9a951ccf95a172f93321cd319 Oct 30 01:17:01.900710 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.900646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:17:01.900834 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.900723 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:01.900834 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.900738 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:17:01.918472 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.918286 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:17:01.918752 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.918729 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wjscp\"" Oct 30 01:17:01.918752 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.918750 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n79rv\"" Oct 30 01:17:01.918893 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.918768 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xp5gx\"" Oct 30 01:17:01.918893 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.918822 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:17:01.939579 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.939560 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:01.959461 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:01.959442 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:02.210908 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.210840 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r" event={"ID":"98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db","Type":"ContainerStarted","Data":"659e8018552c405d80f18629929fddbb026361e9a951ccf95a172f93321cd319"} Oct 30 01:17:02.213834 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.212907 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c9s9t" event={"ID":"4a8853b4-1b14-4b16-81d8-007e7827d8dd","Type":"ContainerStarted","Data":"e420068f5985570ab297321cc6ca305a77e44b7b074f9ecf71bcfd0211723fa2"} Oct 30 01:17:02.213834 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.212962 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-c9s9t" event={"ID":"4a8853b4-1b14-4b16-81d8-007e7827d8dd","Type":"ContainerStarted","Data":"26a4ee28be21d0a49084db02816636f8b892cb1441c9cd007faaedbb29923b4a"} Oct 30 01:17:02.218542 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.218517 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" event={"ID":"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf","Type":"ContainerStarted","Data":"e22fe9cb499d4fedab46c2f49f47021cc3c5614e5e0be3322b6117465b462b08"} Oct 30 01:17:02.224533 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.224469 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lh2vb" event={"ID":"8855c2e7-9182-4ae6-b96a-50baa69d9a67","Type":"ContainerStarted","Data":"49a14e50a6ab971df021c9e1e711e0da45528de7acef663ee026eadb4d2d11a8"} Oct 30 01:17:02.228694 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.228667 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4q66s" event={"ID":"1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5","Type":"ContainerStarted","Data":"417da56a53dd5466875dee9c7d1f226d791a7568076ef320b40a86db326ef251"} Oct 30 01:17:02.228800 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.228706 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-4q66s" event={"ID":"1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5","Type":"ContainerStarted","Data":"b9c6b81d1b651e648a6f8dbe14cccc2c4e2172dd5f3ecc71e9a0deb4fc2a72a8"} Oct 30 01:17:02.235985 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.235929 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-64j9l" event={"ID":"083c7373-177e-46cb-99c1-f9509121668a","Type":"ContainerStarted","Data":"5586bcc51fe3d1f925e6091add497847856a02c5959decadb4c2a0144982ba6e"} Oct 30 01:17:02.237673 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.237618 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" event={"ID":"bb940654-4320-4340-a2d5-9450bc68f40f","Type":"ContainerStarted","Data":"d5d350864092bebeec0e23ee5d30d9483b06aabba3a025eae44cad5a0d3264b3"} Oct 30 01:17:02.239698 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.239651 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv" event={"ID":"495d54bf-6c21-46ab-badd-8ece131d6618","Type":"ContainerStarted","Data":"b3613ef44430a444c8560394f44d0ef7210b83d832738f1907fc15b6d90c131b"} Oct 30 01:17:02.241809 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.241785 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" event={"ID":"712bffb7-97f2-422a-ba7b-cce9eafccba4","Type":"ContainerStarted","Data":"17cb1ef5e6ccb32fd7a4049f9d2901ee7acde9ff5e2b8100424aec55d7637def"} Oct 30 01:17:02.246646 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.246623 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xsnwp" event={"ID":"b4e79600-74c8-4e22-8fce-fbd02aca1e38","Type":"ContainerStarted","Data":"e98cdd7347623e773dbaa6f45a05f19ff1d71b86dd39c061f71de39180b8c498"} Oct 30 01:17:02.246739 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.246662 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xsnwp" event={"ID":"b4e79600-74c8-4e22-8fce-fbd02aca1e38","Type":"ContainerStarted","Data":"dd4f7bdb4814f7c89507166194d8182512d9eae411e8af641cae73ec672b1274"} Oct 30 01:17:02.257804 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.256959 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-c9s9t" podStartSLOduration=427.256945065 podStartE2EDuration="7m7.256945065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:02.256613353 +0000 UTC m=+31.794954295" watchObservedRunningTime="2026-10-30 01:17:02.256945065 +0000 UTC m=+31.795286011" Oct 30 01:17:02.301770 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.301717 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xsnwp" podStartSLOduration=427.301701089 podStartE2EDuration="7m7.301701089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:02.30085364 +0000 UTC m=+31.839194583" watchObservedRunningTime="2026-10-30 01:17:02.301701089 +0000 UTC m=+31.840042031" Oct 30 01:17:02.484352 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.484247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:02.484352 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.484293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:02.484557 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.484384 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:02.484557 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.484416 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:02.484557 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.484496 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:02.484688 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484417 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:17:02.484688 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484684 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54db9d5677-ptwtb: secret "image-registry-tls" not found Oct 30 01:17:02.484787 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484510 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.484491956 +0000 UTC m=+34.022832878 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : configmap references non-existent config key: service-ca.crt Oct 30 01:17:02.484787 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484577 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:17:02.484787 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484736 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655b484568-p27mb: secret "image-registry-tls" not found Oct 30 01:17:02.484787 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484630 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:17:02.484787 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484775 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls podName:806fd3b6-00de-4f42-8c35-e638d3835ec7 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.484758884 +0000 UTC m=+34.023099808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls") pod "image-registry-54db9d5677-ptwtb" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7") : secret "image-registry-tls" not found Oct 30 01:17:02.484787 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484670 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:02.485182 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484801 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls podName:7082f68a-2079-426e-997f-1d3ba3997ba6 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.484792315 +0000 UTC m=+34.023133241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls") pod "image-registry-655b484568-p27mb" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6") : secret "image-registry-tls" not found Oct 30 01:17:02.485182 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484815 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert podName:0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.484807393 +0000 UTC m=+34.023148313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kzdph" (UID: "0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41") : secret "networking-console-plugin-cert" not found Oct 30 01:17:02.485182 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.484840 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:02.485182 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484931 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 01:17:02.485182 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484935 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls podName:c9b752e3-e38a-4d0f-babd-7922d6a2e872 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.484924971 +0000 UTC m=+34.023265897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pb877" (UID: "c9b752e3-e38a-4d0f-babd-7922d6a2e872") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:02.485182 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.484958 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.48494879 +0000 UTC m=+34.023289709 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : secret "router-metrics-certs-default" not found Oct 30 01:17:02.586565 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.586155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:02.586565 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:02.586224 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:02.586565 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.586380 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 01:17:02.586565 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.586435 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls podName:1e24a38d-9f35-40bf-bb90-57fee63d8ac0 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.586416812 +0000 UTC m=+34.124757750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls") pod "dns-default-qdpcp" (UID: "1e24a38d-9f35-40bf-bb90-57fee63d8ac0") : secret "dns-default-metrics-tls" not found Oct 30 01:17:02.586565 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.586497 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 01:17:02.586565 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:02.586534 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls podName:4588e6fe-4b1e-4e62-8528-7411e598b444 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:04.586521139 +0000 UTC m=+34.124862075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54ddb" (UID: "4588e6fe-4b1e-4e62-8528-7411e598b444") : secret "samples-operator-tls" not found Oct 30 01:17:04.256677 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.256604 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r" event={"ID":"98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db","Type":"ContainerStarted","Data":"a33f0c4d254c9232fdebbd0ae64d14384e3ff344b0024e6fd021cf2854f511b5"} Oct 30 01:17:04.300237 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.300179 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-4q66s" podStartSLOduration=429.300160891 podStartE2EDuration="7m9.300160891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:02.347284638 +0000 UTC m=+31.885625579" watchObservedRunningTime="2026-10-30 01:17:04.300160891 +0000 UTC m=+33.838501833" Oct 30 01:17:04.508832 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.508748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:04.508832 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.508799 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:04.509086 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.508848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:04.509086 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.508870 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:04.509086 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.508926 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:04.509086 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.508930 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:17:04.509086 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.508954 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54db9d5677-ptwtb: secret "image-registry-tls" not found Oct 30 01:17:04.509086 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509007 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls podName:806fd3b6-00de-4f42-8c35-e638d3835ec7 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:08.508991958 +0000 UTC m=+38.047332894 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls") pod "image-registry-54db9d5677-ptwtb" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7") : secret "image-registry-tls" not found Oct 30 01:17:04.509086 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509010 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 01:17:04.509086 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509048 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:08.509036968 +0000 UTC m=+38.047377888 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : secret "router-metrics-certs-default" not found Oct 30 01:17:04.509495 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509104 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:08.509097454 +0000 UTC m=+38.047438410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : configmap references non-existent config key: service-ca.crt Oct 30 01:17:04.509495 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509147 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:17:04.509495 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509168 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert podName:0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:08.50916191 +0000 UTC m=+38.047502831 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kzdph" (UID: "0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41") : secret "networking-console-plugin-cert" not found Oct 30 01:17:04.509495 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509201 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:04.509495 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509225 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls podName:c9b752e3-e38a-4d0f-babd-7922d6a2e872 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:08.509216861 +0000 UTC m=+38.047557781 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pb877" (UID: "c9b752e3-e38a-4d0f-babd-7922d6a2e872") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:04.509495 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509263 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:17:04.509495 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509270 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655b484568-p27mb: secret "image-registry-tls" not found Oct 30 01:17:04.509495 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.509293 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls podName:7082f68a-2079-426e-997f-1d3ba3997ba6 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:08.509283944 +0000 UTC m=+38.047624905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls") pod "image-registry-655b484568-p27mb" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6") : secret "image-registry-tls" not found Oct 30 01:17:04.509495 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.508942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:04.610645 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.610606 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:04.610812 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.610682 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:04.610948 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.610933 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 01:17:04.611011 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.610992 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls podName:1e24a38d-9f35-40bf-bb90-57fee63d8ac0 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:08.610975371 +0000 UTC m=+38.149316313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls") pod "dns-default-qdpcp" (UID: "1e24a38d-9f35-40bf-bb90-57fee63d8ac0") : secret "dns-default-metrics-tls" not found Oct 30 01:17:04.611078 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.611057 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 01:17:04.611125 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.611091 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls podName:4588e6fe-4b1e-4e62-8528-7411e598b444 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:08.611080865 +0000 UTC m=+38.149421803 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54ddb" (UID: "4588e6fe-4b1e-4e62-8528-7411e598b444") : secret "samples-operator-tls" not found Oct 30 01:17:04.712229 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.712190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:04.712431 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.712369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:17:04.712566 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.712539 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 01:17:04.712696 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:04.712607 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs podName:d8c54d89-eea4-45ee-80e5-fd0aba53e274 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:36.712587924 +0000 UTC m=+66.250928860 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs") pod "network-metrics-daemon-p9qrg" (UID: "d8c54d89-eea4-45ee-80e5-fd0aba53e274") : secret "metrics-daemon-secret" not found Oct 30 01:17:04.715655 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.715629 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:04.813580 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.813294 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:17:04.826675 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.826648 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:17:04.948219 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.948184 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:17:04.971414 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:04.971173 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:05.291977 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:05.291932 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r" podStartSLOduration=386.937989143 podStartE2EDuration="6m29.29191431s" totalImagesPullingTime="2.353925167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:04.301708665 +0000 UTC m=+33.840049608" watchObservedRunningTime="2026-10-30 01:17:05.29191431 +0000 UTC m=+34.830255255" Oct 30 01:17:05.293419 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:05.293293 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4cnv8"] Oct 30 01:17:05.328582 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:05.328556 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rz57q"] Oct 30 01:17:05.402816 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:05.402759 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf09128_a8c1_4d32_958b_cbd01b892097.slice/crio-a0da20c9274c1300ff32558da7553589ed351a415146668daef151225ec05b13 WatchSource:0}: Error finding container a0da20c9274c1300ff32558da7553589ed351a415146668daef151225ec05b13: Status 404 returned error can't find the container with id a0da20c9274c1300ff32558da7553589ed351a415146668daef151225ec05b13 Oct 30 01:17:06.264933 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:06.264894 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rz57q" event={"ID":"8bc436c0-2b1b-477e-81aa-af1a1eff9779","Type":"ContainerStarted","Data":"1706b219852576de48023ea1f2304cbae21c23bccbabd5de5f54a3168faf15a4"} Oct 30 01:17:06.266875 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:06.266841 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4cnv8" event={"ID":"dbf09128-a8c1-4d32-958b-cbd01b892097","Type":"ContainerStarted","Data":"a0da20c9274c1300ff32558da7553589ed351a415146668daef151225ec05b13"} Oct 30 01:17:06.268828 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:06.268807 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/0.log" Oct 30 01:17:06.268930 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:06.268847 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" containerID="5a6fdefa22343420eea3e1441f0927a6c3c13e7f2c7a8f350e7f3c0eeb338799" exitCode=255 Oct 30 01:17:06.268930 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:06.268885 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" event={"ID":"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf","Type":"ContainerDied","Data":"5a6fdefa22343420eea3e1441f0927a6c3c13e7f2c7a8f350e7f3c0eeb338799"} Oct 30 01:17:06.269171 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:06.269150 2578 scope.go:122] "RemoveContainer" containerID="5a6fdefa22343420eea3e1441f0927a6c3c13e7f2c7a8f350e7f3c0eeb338799" Oct 30 01:17:07.213978 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.213943 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jz2gt"] Oct 30 01:17:07.232065 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.232042 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.237234 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.237175 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jz2gt"] Oct 30 01:17:07.250602 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.250551 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:17:07.250706 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.250581 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:17:07.250706 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.250652 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xmk59\"" Oct 30 01:17:07.285658 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.285627 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lh2vb" event={"ID":"8855c2e7-9182-4ae6-b96a-50baa69d9a67","Type":"ContainerStarted","Data":"2a884be7e6b2d077c9a69df98ac80c8efead0549836a837e8dab20fc818a7a32"} Oct 30 01:17:07.287493 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.287470 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-gmfkz-7g8gt"] Oct 30 01:17:07.295847 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.295824 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.295999 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.295963 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-64j9l" event={"ID":"083c7373-177e-46cb-99c1-f9509121668a","Type":"ContainerStarted","Data":"c02f7ece88e3f27ea8d6f5775cc36faa3c688969f91108c83f3f95744c7ddea3"} Oct 30 01:17:07.299351 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.299305 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" event={"ID":"bb940654-4320-4340-a2d5-9450bc68f40f","Type":"ContainerStarted","Data":"674cb174aefe2d90a87f442845e6021a0c7dccaf30de5c8b7871a5320eb3b494"} Oct 30 01:17:07.302544 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.302479 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" event={"ID":"712bffb7-97f2-422a-ba7b-cce9eafccba4","Type":"ContainerStarted","Data":"51d87176a0f58d0c89903dad4b40b47490319828a6787d6294139d0e2887c900"} Oct 30 01:17:07.306876 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.306783 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/1.log" Oct 30 01:17:07.307450 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.307368 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/0.log" Oct 30 01:17:07.307742 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.307542 2578 generic.go:356] "Generic (PLEG): container finished" podID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" containerID="d3e27a8324376e2d5c186fcfa676d1a7925a6e551c97b7f07801c298375c6438" exitCode=255 Oct 30 01:17:07.307742 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.307590 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" event={"ID":"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf","Type":"ContainerDied","Data":"d3e27a8324376e2d5c186fcfa676d1a7925a6e551c97b7f07801c298375c6438"} Oct 30 01:17:07.307742 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.307629 2578 scope.go:122] "RemoveContainer" containerID="5a6fdefa22343420eea3e1441f0927a6c3c13e7f2c7a8f350e7f3c0eeb338799" Oct 30 01:17:07.307742 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.307703 2578 scope.go:122] "RemoveContainer" containerID="d3e27a8324376e2d5c186fcfa676d1a7925a6e551c97b7f07801c298375c6438" Oct 30 01:17:07.307957 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:07.307896 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tw27b_openshift-console-operator(b6cb0d17-b098-408e-92b6-b5e81a8bb9cf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" podUID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" Oct 30 01:17:07.308318 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.308297 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gmfkz-7g8gt"] Oct 30 01:17:07.348410 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.347992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.348410 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.348029 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf4dt\" (UniqueName: \"kubernetes.io/projected/184de02f-0273-408d-9ee5-265a0bb278a4-kube-api-access-mf4dt\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.348410 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.348080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/184de02f-0273-408d-9ee5-265a0bb278a4-data-volume\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.348410 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.348152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.348410 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.348242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/184de02f-0273-408d-9ee5-265a0bb278a4-crio-socket\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.388217 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.388162 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-64j9l" podStartSLOduration=386.620874719 podStartE2EDuration="6m31.388148063s" totalImagesPullingTime="4.767273344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:07.38740827 +0000 UTC m=+36.925749212" watchObservedRunningTime="2026-10-30 01:17:07.388148063 +0000 UTC m=+36.926489064" Oct 30 01:17:07.436046 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.435986 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lh2vb" podStartSLOduration=2.256904831 podStartE2EDuration="7.43596731s" totalImagesPullingTime="5.179062479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:07.43420534 +0000 UTC m=+36.972546285" watchObservedRunningTime="2026-10-30 01:17:07.43596731 +0000 UTC m=+36.974308253" Oct 30 01:17:07.449596 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.449549 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/184de02f-0273-408d-9ee5-265a0bb278a4-data-volume\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.449596 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.449592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-archives-path\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.450024 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.450001 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-trusted-ca-bundle\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.450096 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.450058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgmg\" (UniqueName: \"kubernetes.io/projected/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-kube-api-access-7vgmg\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.450223 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.450201 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.450481 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.450455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/184de02f-0273-408d-9ee5-265a0bb278a4-crio-socket\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.450557 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.450541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.450620 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.450573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4dt\" (UniqueName: \"kubernetes.io/projected/184de02f-0273-408d-9ee5-265a0bb278a4-kube-api-access-mf4dt\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.450662 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.450652 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-service-ca-bundle\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.451027 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.451001 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/184de02f-0273-408d-9ee5-265a0bb278a4-data-volume\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.452073 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:07.451577 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 01:17:07.452073 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:07.451640 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:07.951620282 +0000 UTC m=+37.489961214 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : configmap "kube-rbac-proxy" not found Oct 30 01:17:07.452073 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:07.451987 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:17:07.452073 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:07.452057 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:07.952037895 +0000 UTC m=+37.490378838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : secret "insights-runtime-extractor-tls" not found Oct 30 01:17:07.452276 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.452263 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/184de02f-0273-408d-9ee5-265a0bb278a4-crio-socket\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.477275 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.477245 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4dt\" (UniqueName: \"kubernetes.io/projected/184de02f-0273-408d-9ee5-265a0bb278a4-kube-api-access-mf4dt\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.480668 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.480615 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" podStartSLOduration=383.647672856 podStartE2EDuration="6m28.480595573s" totalImagesPullingTime="4.832922717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:07.479133796 +0000 UTC m=+37.017474739" watchObservedRunningTime="2026-10-30 01:17:07.480595573 +0000 UTC m=+37.018936516" Oct 30 01:17:07.552867 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.551967 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-archives-path\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.552867 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.552022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-trusted-ca-bundle\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.552867 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.552059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgmg\" (UniqueName: \"kubernetes.io/projected/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-kube-api-access-7vgmg\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.552867 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.552289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-service-ca-bundle\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.552867 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.552399 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-archives-path\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.552867 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.552825 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-service-ca-bundle\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.553161 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.553079 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-trusted-ca-bundle\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.581655 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.581598 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgmg\" (UniqueName: \"kubernetes.io/projected/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-kube-api-access-7vgmg\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.620348 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.616813 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" podStartSLOduration=391.780631327 podStartE2EDuration="6m36.616786467s" totalImagesPullingTime="4.83615514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:07.582593858 +0000 UTC m=+37.120934800" watchObservedRunningTime="2026-10-30 01:17:07.616786467 +0000 UTC m=+37.155127410" Oct 30 01:17:07.620348 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.619294 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm"] Oct 30 01:17:07.628039 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.628012 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:07.642353 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.639124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm" Oct 30 01:17:07.655729 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.655543 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26"] Oct 30 01:17:07.675606 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.675030 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-g5b5q\"" Oct 30 01:17:07.677637 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.677346 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:07.677637 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.677575 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 01:17:07.683925 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.683445 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm"] Oct 30 01:17:07.683925 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.683561 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26" Oct 30 01:17:07.714428 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.713967 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26"] Oct 30 01:17:07.754506 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.754425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjxrh\" (UniqueName: \"kubernetes.io/projected/56fd7c23-5b71-424f-87e6-91f34918efe8-kube-api-access-fjxrh\") pod \"migrator-66df5d496b-jj7dm\" (UID: \"56fd7c23-5b71-424f-87e6-91f34918efe8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm" Oct 30 01:17:07.856511 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.855986 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxrh\" (UniqueName: \"kubernetes.io/projected/56fd7c23-5b71-424f-87e6-91f34918efe8-kube-api-access-fjxrh\") pod \"migrator-66df5d496b-jj7dm\" (UID: \"56fd7c23-5b71-424f-87e6-91f34918efe8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm" Oct 30 01:17:07.856511 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.856102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4d5n\" (UniqueName: \"kubernetes.io/projected/cc0b4765-071a-4af8-88e6-700abd1d8f83-kube-api-access-k4d5n\") pod \"migrator-66df5d496b-jgj26\" (UID: \"cc0b4765-071a-4af8-88e6-700abd1d8f83\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26" Oct 30 01:17:07.891873 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.891844 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxrh\" (UniqueName: \"kubernetes.io/projected/56fd7c23-5b71-424f-87e6-91f34918efe8-kube-api-access-fjxrh\") pod \"migrator-66df5d496b-jj7dm\" (UID: \"56fd7c23-5b71-424f-87e6-91f34918efe8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm" Oct 30 01:17:07.957398 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.957371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4d5n\" (UniqueName: \"kubernetes.io/projected/cc0b4765-071a-4af8-88e6-700abd1d8f83-kube-api-access-k4d5n\") pod \"migrator-66df5d496b-jgj26\" (UID: \"cc0b4765-071a-4af8-88e6-700abd1d8f83\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26" Oct 30 01:17:07.957536 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.957434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.957579 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.957541 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:07.957669 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:07.957652 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 01:17:07.957732 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:07.957716 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:08.957697963 +0000 UTC m=+38.496038897 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : configmap "kube-rbac-proxy" not found Oct 30 01:17:07.958601 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:07.958578 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:17:07.958702 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:07.958647 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:08.958629048 +0000 UTC m=+38.496969983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : secret "insights-runtime-extractor-tls" not found Oct 30 01:17:07.979853 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.979830 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-gmfkz-7g8gt"] Oct 30 01:17:07.980563 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:07.980535 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc585a4_2d42_40f1_bd8d_c17d9dc7f591.slice/crio-6b3ff3a129302e0236657cddceb47cb1094f1e633f34692359e065d77cd2a394 WatchSource:0}: Error finding container 6b3ff3a129302e0236657cddceb47cb1094f1e633f34692359e065d77cd2a394: Status 404 returned error can't find the container with id 6b3ff3a129302e0236657cddceb47cb1094f1e633f34692359e065d77cd2a394 Oct 30 01:17:07.989779 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.989716 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4d5n\" (UniqueName: \"kubernetes.io/projected/cc0b4765-071a-4af8-88e6-700abd1d8f83-kube-api-access-k4d5n\") pod \"migrator-66df5d496b-jgj26\" (UID: \"cc0b4765-071a-4af8-88e6-700abd1d8f83\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26" Oct 30 01:17:07.998115 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:07.997935 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm" Oct 30 01:17:08.020213 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.020151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26" Oct 30 01:17:08.092588 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.091111 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2c2hh"] Oct 30 01:17:08.118732 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.118693 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2c2hh"] Oct 30 01:17:08.118870 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.118808 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.136837 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.136611 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 01:17:08.136837 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.136716 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 01:17:08.136837 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.136809 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5t49s\"" Oct 30 01:17:08.142171 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.142151 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 01:17:08.144353 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.144308 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 01:17:08.261072 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.261037 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7cmp\" (UniqueName: \"kubernetes.io/projected/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-kube-api-access-h7cmp\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.261537 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.261126 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-key\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.261537 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.261239 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-cabundle\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.314458 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.314366 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv" event={"ID":"495d54bf-6c21-46ab-badd-8ece131d6618","Type":"ContainerStarted","Data":"5c3b5c00762c5a3eccdeca63a39ad6e38b1b2d9cf623b4c7565ae816a3f9f7e0"} Oct 30 01:17:08.318361 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.317467 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/1.log" Oct 30 01:17:08.318361 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.317855 2578 scope.go:122] "RemoveContainer" containerID="d3e27a8324376e2d5c186fcfa676d1a7925a6e551c97b7f07801c298375c6438" Oct 30 01:17:08.318361 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.318056 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tw27b_openshift-console-operator(b6cb0d17-b098-408e-92b6-b5e81a8bb9cf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" podUID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" Oct 30 01:17:08.320374 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.320286 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" event={"ID":"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591","Type":"ContainerStarted","Data":"357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188"} Oct 30 01:17:08.320374 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.320345 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" event={"ID":"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591","Type":"ContainerStarted","Data":"6b3ff3a129302e0236657cddceb47cb1094f1e633f34692359e065d77cd2a394"} Oct 30 01:17:08.344507 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.344468 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm"] Oct 30 01:17:08.347678 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:08.347638 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56fd7c23_5b71_424f_87e6_91f34918efe8.slice/crio-bd2f1bbed8bd0f04e30ec784b0bfcab605d5f52815145ddab8441674fabcc6ca WatchSource:0}: Error finding container bd2f1bbed8bd0f04e30ec784b0bfcab605d5f52815145ddab8441674fabcc6ca: Status 404 returned error can't find the container with id bd2f1bbed8bd0f04e30ec784b0bfcab605d5f52815145ddab8441674fabcc6ca Oct 30 01:17:08.362838 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.362769 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7cmp\" (UniqueName: \"kubernetes.io/projected/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-kube-api-access-h7cmp\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.363259 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.363003 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-key\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.363259 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.363124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-cabundle\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.364132 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.364077 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-cabundle\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.366722 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.366673 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-key\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.385384 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.385361 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7cmp\" (UniqueName: \"kubernetes.io/projected/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-kube-api-access-h7cmp\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.408204 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.408151 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv" podStartSLOduration=383.462267025 podStartE2EDuration="6m29.408137147s" totalImagesPullingTime="5.945870122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:08.362134101 +0000 UTC m=+37.900475043" watchObservedRunningTime="2026-10-30 01:17:08.408137147 +0000 UTC m=+37.946478091" Oct 30 01:17:08.417763 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.417693 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26"] Oct 30 01:17:08.418998 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:08.418977 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0b4765_071a_4af8_88e6_700abd1d8f83.slice/crio-e5e9d78a986a1f47384efa0f900544bb1293a10a69b01eb62f365e40887c82e6 WatchSource:0}: Error finding container e5e9d78a986a1f47384efa0f900544bb1293a10a69b01eb62f365e40887c82e6: Status 404 returned error can't find the container with id e5e9d78a986a1f47384efa0f900544bb1293a10a69b01eb62f365e40887c82e6 Oct 30 01:17:08.449730 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.449706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:08.452176 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.452080 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" podStartSLOduration=1.452065448 podStartE2EDuration="1.452065448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:08.450125503 +0000 UTC m=+37.988466446" watchObservedRunningTime="2026-10-30 01:17:08.452065448 +0000 UTC m=+37.990406413" Oct 30 01:17:08.566011 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.565942 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:08.566011 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.565993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:08.566428 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.566059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:08.566428 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.566085 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:08.566428 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566178 2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:08.566428 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566191 2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 01:17:08.566428 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566249 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls podName:c9b752e3-e38a-4d0f-babd-7922d6a2e872 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:16.566229809 +0000 UTC m=+46.104570749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pb877" (UID: "c9b752e3-e38a-4d0f-babd-7922d6a2e872") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:08.566428 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.566184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:08.566428 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566270 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:17:08.566428 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566285 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-54db9d5677-ptwtb: secret "image-registry-tls" not found Oct 30 01:17:08.566428 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.566317 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:08.566921 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566477 2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 01:17:08.566921 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566489 2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-655b484568-p27mb: secret "image-registry-tls" not found Oct 30 01:17:08.566921 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566507 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls podName:806fd3b6-00de-4f42-8c35-e638d3835ec7 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:16.56649169 +0000 UTC m=+46.104832625 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls") pod "image-registry-54db9d5677-ptwtb" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7") : secret "image-registry-tls" not found Oct 30 01:17:08.566921 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566527 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls podName:7082f68a-2079-426e-997f-1d3ba3997ba6 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:16.566517318 +0000 UTC m=+46.104858264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls") pod "image-registry-655b484568-p27mb" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6") : secret "image-registry-tls" not found Oct 30 01:17:08.566921 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566554 2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 01:17:08.566921 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566590 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:16.566572878 +0000 UTC m=+46.104913813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : secret "router-metrics-certs-default" not found Oct 30 01:17:08.566921 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566656 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:16.566641408 +0000 UTC m=+46.104982331 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : configmap references non-existent config key: service-ca.crt Oct 30 01:17:08.566921 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.566795 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert podName:0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:16.566780332 +0000 UTC m=+46.105121259 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-kzdph" (UID: "0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41") : secret "networking-console-plugin-cert" not found Oct 30 01:17:08.667430 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.667385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:08.667580 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.667457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:08.667580 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.667533 2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 01:17:08.667580 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.667546 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 01:17:08.667695 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.667589 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls podName:1e24a38d-9f35-40bf-bb90-57fee63d8ac0 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:16.667577763 +0000 UTC m=+46.205918687 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls") pod "dns-default-qdpcp" (UID: "1e24a38d-9f35-40bf-bb90-57fee63d8ac0") : secret "dns-default-metrics-tls" not found Oct 30 01:17:08.667695 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.667601 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls podName:4588e6fe-4b1e-4e62-8528-7411e598b444 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:16.667596049 +0000 UTC m=+46.205936968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-54ddb" (UID: "4588e6fe-4b1e-4e62-8528-7411e598b444") : secret "samples-operator-tls" not found Oct 30 01:17:08.801820 ip-10-0-134-254 kubenswrapper[2578]: W1030 01:17:08.801788 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4577e2_2c6f_4faf_a425_4ce3e00cec84.slice/crio-16dae0468c0af28ebfa64b54be4527d564599feec177245e244bfb390916d6c1 WatchSource:0}: Error finding container 16dae0468c0af28ebfa64b54be4527d564599feec177245e244bfb390916d6c1: Status 404 returned error can't find the container with id 16dae0468c0af28ebfa64b54be4527d564599feec177245e244bfb390916d6c1 Oct 30 01:17:08.802461 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.802439 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-2c2hh"] Oct 30 01:17:08.970071 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.970033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:08.970279 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:08.970150 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:08.970279 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.970159 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:17:08.970279 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.970224 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:10.97020528 +0000 UTC m=+40.508546216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : secret "insights-runtime-extractor-tls" not found Oct 30 01:17:08.970540 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.970275 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 01:17:08.970540 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:08.970366 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:10.970321644 +0000 UTC m=+40.508662576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : configmap "kube-rbac-proxy" not found Oct 30 01:17:09.345267 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:09.344756 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" event={"ID":"9f4577e2-2c6f-4faf-a425-4ce3e00cec84","Type":"ContainerStarted","Data":"d6548acb6c07e61fe5ede7ab8787b07cd30056d004523823a51b1210e587c041"} Oct 30 01:17:09.345267 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:09.344824 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" event={"ID":"9f4577e2-2c6f-4faf-a425-4ce3e00cec84","Type":"ContainerStarted","Data":"16dae0468c0af28ebfa64b54be4527d564599feec177245e244bfb390916d6c1"} Oct 30 01:17:09.348121 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:09.348090 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4cnv8" event={"ID":"dbf09128-a8c1-4d32-958b-cbd01b892097","Type":"ContainerStarted","Data":"01e5658b0ad6da1cd4aaeccf2c1fa3e80d11795232a6b0c127f7f531e6c7a9b9"} Oct 30 01:17:09.348253 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:09.348173 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:17:09.349946 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:09.349918 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26" event={"ID":"cc0b4765-071a-4af8-88e6-700abd1d8f83","Type":"ContainerStarted","Data":"e5e9d78a986a1f47384efa0f900544bb1293a10a69b01eb62f365e40887c82e6"} Oct 30 01:17:09.351377 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:09.351296 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm" event={"ID":"56fd7c23-5b71-424f-87e6-91f34918efe8","Type":"ContainerStarted","Data":"bd2f1bbed8bd0f04e30ec784b0bfcab605d5f52815145ddab8441674fabcc6ca"} Oct 30 01:17:09.402661 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:09.402609 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" podStartSLOduration=1.4025909730000001 podStartE2EDuration="1.402590973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:09.400697358 +0000 UTC m=+38.939038301" watchObservedRunningTime="2026-10-30 01:17:09.402590973 +0000 UTC m=+38.940931917" Oct 30 01:17:09.449773 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:09.449715 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4cnv8" podStartSLOduration=35.066703574 podStartE2EDuration="38.449703933s" totalImagesPullingTime="3.383000359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:09.448573682 +0000 UTC m=+38.986914625" watchObservedRunningTime="2026-10-30 01:17:09.449703933 +0000 UTC m=+38.988044874" Oct 30 01:17:10.991746 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:10.991717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:10.992225 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:10.991831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:10.992225 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:10.991871 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:17:10.992225 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:10.991934 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:14.99191593 +0000 UTC m=+44.530256872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : secret "insights-runtime-extractor-tls" not found Oct 30 01:17:10.992225 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:10.991970 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 01:17:10.992225 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:10.992042 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:14.992023651 +0000 UTC m=+44.530364589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : configmap "kube-rbac-proxy" not found Oct 30 01:17:11.087224 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:11.087190 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:11.087393 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:11.087245 2578 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:11.087648 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:11.087625 2578 scope.go:122] "RemoveContainer" containerID="d3e27a8324376e2d5c186fcfa676d1a7925a6e551c97b7f07801c298375c6438" Oct 30 01:17:11.087845 ip-10-0-134-254 kubenswrapper[2578]: E1030 01:17:11.087825 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tw27b_openshift-console-operator(b6cb0d17-b098-408e-92b6-b5e81a8bb9cf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" podUID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" Oct 30 01:17:11.360212 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:11.360127 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26" event={"ID":"cc0b4765-071a-4af8-88e6-700abd1d8f83","Type":"ContainerStarted","Data":"2bb1aa4bc2eedaf08c9caedf8df0139542b1f4df6a5ab8547580dc48a08d051c"} Oct 30 01:17:11.362131 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:11.362101 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm" event={"ID":"56fd7c23-5b71-424f-87e6-91f34918efe8","Type":"ContainerStarted","Data":"ca7cdd5386a3d6675e4ffd81946522e95547b1ea2422b8f7a7b6cfef439f5ecc"} Oct 30 01:17:11.404196 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:11.404142 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26" podStartSLOduration=2.496563708 podStartE2EDuration="4.4041292s" totalImagesPullingTime="1.907565492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:11.403970101 +0000 UTC m=+40.942311043" watchObservedRunningTime="2026-10-30 01:17:11.4041292 +0000 UTC m=+40.942470142" Oct 30 01:17:11.448046 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:11.447999 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm" podStartSLOduration=2.468298779 podStartE2EDuration="4.447986514s" totalImagesPullingTime="1.979687735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:11.447117869 +0000 UTC m=+40.985458827" watchObservedRunningTime="2026-10-30 01:17:11.447986514 +0000 UTC m=+40.986327456" Oct 30 01:17:12.618936 ip-10-0-134-254 kubenswrapper[2578]: I1030 01:17:12.618707 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:17:12.618872 ip-10-0-134-254 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 01:17:12.632712 ip-10-0-134-254 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 01:17:12.632887 ip-10-0-134-254 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 01:17:12.632933 ip-10-0-134-254 systemd[1]: kubelet.service: Consumed 3.171s CPU time. Oct 30 01:17:12.647789 ip-10-0-134-254 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:17:12.738525 ip-10-0-134-254 kubenswrapper[5055]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:17:12.738525 ip-10-0-134-254 kubenswrapper[5055]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:17:12.738525 ip-10-0-134-254 kubenswrapper[5055]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:17:12.738525 ip-10-0-134-254 kubenswrapper[5055]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:17:12.740570 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740494 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:12.740570 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740568 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:12.740570 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740573 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:12.740570 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740577 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740580 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740583 5055 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740586 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740589 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740593 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740602 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740606 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740610 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740614 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740618 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740624 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740629 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740632 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740635 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740638 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740641 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740643 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740646 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:12.740717 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740649 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740651 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740654 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740657 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740659 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740662 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740666 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740671 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740674 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740677 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740680 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740683 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740685 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740688 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740691 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740694 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740696 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740699 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740702 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:12.741202 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740704 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740707 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740709 5055 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740712 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740714 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740717 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740720 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740722 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740725 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740728 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740730 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740733 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740735 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740738 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740741 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740744 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740747 5055 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740750 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740753 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740755 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:12.741701 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740758 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740761 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740763 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740766 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740768 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740772 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740774 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740777 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740780 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740783 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740785 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740789 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740792 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740794 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740797 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740800 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740803 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740805 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740808 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:12.742204 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740810 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740813 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740815 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740818 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740821 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740825 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740827 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740830 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740833 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740836 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740840 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740842 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740845 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740848 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740851 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740853 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740855 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740858 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740861 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740863 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:12.742680 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740866 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740869 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740871 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740874 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740876 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.740882 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741701 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741710 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741714 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741718 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741721 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741723 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741726 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741729 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741731 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741734 5055 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741737 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741739 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741742 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:12.743186 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741744 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741747 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741749 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741752 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741754 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741757 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741759 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741762 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741764 5055 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741767 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741772 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741776 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741779 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741781 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741784 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741787 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741789 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741792 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741795 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741798 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:12.743686 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741801 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741804 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741807 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741809 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741812 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741815 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741817 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741820 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741823 5055 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741826 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741828 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741831 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741833 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741848 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741852 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741856 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741859 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741862 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741865 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:12.744208 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741868 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741871 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741874 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741876 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741879 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741881 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741884 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741886 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741889 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741891 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741894 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741897 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741899 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741902 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741905 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741908 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741910 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741913 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741916 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:12.744692 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741919 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741922 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741924 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741927 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741929 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741932 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741934 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741938 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741941 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741944 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741947 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741949 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741952 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741954 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741957 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741959 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741962 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741964 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741967 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:12.745152 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741969 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741972 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741974 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741977 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741979 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741982 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741986 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741989 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741992 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741995 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.741997 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742000 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742003 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742006 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742008 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742012 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742493 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742499 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742502 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742504 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:12.745676 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742508 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742510 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742513 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742516 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742518 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742521 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742524 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742527 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742529 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742532 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742534 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742537 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742539 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742542 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742545 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742548 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742551 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742553 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742558 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742561 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:12.746161 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742564 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742567 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742570 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742572 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742576 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742579 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742582 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742584 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742587 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742592 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742596 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742599 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742602 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742605 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742607 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742610 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742613 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742615 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742618 5055 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:12.746652 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742621 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742623 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742626 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742628 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742631 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742633 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742636 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742638 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742641 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742643 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742646 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742650 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742652 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742656 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742660 5055 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742664 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742666 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742672 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742675 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:12.747109 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742678 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742681 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742684 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742687 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742689 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742692 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742695 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742698 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742701 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742704 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742706 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742709 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742712 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742714 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742717 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742719 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742722 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742724 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742727 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742730 5055 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:12.747585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742732 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742735 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742737 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742740 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742743 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742746 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742749 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742751 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742754 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742757 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742761 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742764 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742767 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742769 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742772 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742774 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742777 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742781 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742784 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742786 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:12.748064 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742789 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742792 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742794 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.742797 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742911 5055 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742921 5055 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742929 5055 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742933 5055 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742937 5055 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742941 5055 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742945 5055 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742950 5055 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742953 5055 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742956 5055 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742959 5055 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742963 5055 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742966 5055 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742969 5055 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742972 5055 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742975 5055 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742979 5055 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742982 5055 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742988 5055 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:17:12.748566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742991 5055 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742995 5055 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.742998 5055 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743001 5055 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743005 5055 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743008 5055 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743012 5055 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743015 5055 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743018 5055 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743021 5055 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743024 5055 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743027 5055 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743030 5055 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743034 5055 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743037 5055 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743040 5055 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743044 5055 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743047 5055 flags.go:64] FLAG: --enable-server="true" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743050 5055 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743055 5055 flags.go:64] FLAG: --event-burst="100" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743058 5055 flags.go:64] FLAG: --event-qps="50" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743061 5055 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743064 5055 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743067 5055 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:17:12.749118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743072 5055 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743075 5055 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743078 5055 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743081 5055 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743084 5055 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743086 5055 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743090 5055 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743094 5055 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743098 5055 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743101 5055 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743104 5055 flags.go:64] FLAG: --feature-gates="" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743107 5055 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743111 5055 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743114 5055 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743117 5055 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743120 5055 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743123 5055 flags.go:64] FLAG: --help="false" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743126 5055 flags.go:64] FLAG: --hostname-override="ip-10-0-134-254.ec2.internal" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743130 5055 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743133 5055 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743136 5055 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743139 5055 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743143 5055 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743145 5055 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:17:12.749721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743148 5055 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743151 5055 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743154 5055 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743157 5055 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743160 5055 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743163 5055 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743166 5055 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743168 5055 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743171 5055 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743174 5055 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743177 5055 flags.go:64] FLAG: --lock-file="" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743180 5055 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743183 5055 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743186 5055 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743191 5055 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743196 5055 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743200 5055 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743203 5055 flags.go:64] FLAG: --logging-format="text" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743206 5055 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743209 5055 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743212 5055 flags.go:64] FLAG: --manifest-url="" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743215 5055 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743219 5055 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743222 5055 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743226 5055 flags.go:64] FLAG: --max-pods="110" Oct 30 01:17:12.750318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743229 5055 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743231 5055 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743234 5055 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743240 5055 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743243 5055 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743245 5055 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743248 5055 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743255 5055 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743258 5055 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743262 5055 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743265 5055 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743268 5055 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743271 5055 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743274 5055 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743277 5055 flags.go:64] FLAG: --port="10250" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743280 5055 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743282 5055 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01aab64cde907bb97" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743286 5055 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743289 5055 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743292 5055 flags.go:64] FLAG: --register-node="true" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743294 5055 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743298 5055 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743303 5055 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743306 5055 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743309 5055 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743312 5055 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:17:12.751173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743316 5055 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743319 5055 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743335 5055 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743338 5055 flags.go:64] FLAG: --runonce="false" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743341 5055 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743344 5055 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743347 5055 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743350 5055 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743353 5055 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743358 5055 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743361 5055 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743364 5055 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743367 5055 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743370 5055 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743373 5055 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743376 5055 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743379 5055 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743382 5055 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743384 5055 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743390 5055 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743393 5055 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743395 5055 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743399 5055 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743402 5055 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743405 5055 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:17:12.752151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743408 5055 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743410 5055 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743413 5055 flags.go:64] FLAG: --v="2" Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743419 5055 flags.go:64] FLAG: --version="false" Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743423 5055 flags.go:64] FLAG: --vmodule="" Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743427 5055 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743430 5055 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743571 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743576 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743579 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743582 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743585 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743588 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743591 5055 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743594 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743597 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743601 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743603 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743606 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743619 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743623 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:12.752904 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743626 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743630 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743633 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743636 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743638 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743641 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743645 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743648 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743652 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743655 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743657 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743660 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743663 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743665 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743670 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743672 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743675 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743677 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743681 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:12.753677 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743683 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743686 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743689 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743691 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743694 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743697 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743700 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743702 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743706 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743709 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743712 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743714 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743717 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743719 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743722 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743724 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743727 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743729 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743732 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:12.754227 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743735 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743737 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743740 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743742 5055 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743745 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743747 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743750 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743752 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743755 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743758 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743760 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743763 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743765 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743768 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743771 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743773 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743776 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743778 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743781 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743783 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:12.754849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743786 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743790 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743792 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743795 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743797 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743799 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743802 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743804 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743807 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743809 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743812 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743814 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743817 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743820 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743824 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743827 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743830 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743832 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743835 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:12.755370 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743837 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743840 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743843 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743845 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743848 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743850 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743853 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743856 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743858 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743861 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743863 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743866 5055 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743868 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743871 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743875 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.743901 5055 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743950 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:12.755907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743955 5055 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743958 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743961 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743963 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743966 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743968 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743971 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743974 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743976 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743978 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743981 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743984 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743986 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743989 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743993 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.743997 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744000 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744003 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:12.756466 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744005 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744008 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744011 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744013 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744016 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744019 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744021 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744024 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744026 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744029 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744031 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744036 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744038 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744041 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744043 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744046 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744048 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744051 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744053 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:12.757102 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744056 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744059 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744061 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744064 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744066 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744069 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744072 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744074 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744077 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744079 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744082 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744084 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744087 5055 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744089 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744092 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744094 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744098 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744100 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744103 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744106 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:12.757837 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744109 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744111 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744114 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744116 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744120 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744123 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744126 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744128 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744131 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744133 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744136 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744138 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744140 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744144 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744146 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744149 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744151 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744154 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744157 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744159 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:12.758315 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744161 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744164 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744168 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744172 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744175 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744177 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744180 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744182 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744186 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744189 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744192 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744194 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744197 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744200 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744214 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744218 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744222 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744225 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744227 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744230 5055 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:12.758823 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744233 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744235 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744238 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744241 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744244 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744247 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744249 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.744252 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.747955 5055 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.747970 5055 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.747986 5055 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.747991 5055 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748046 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748051 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748054 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748057 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748060 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748062 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748066 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:12.759320 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748068 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748072 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748077 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748080 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748083 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748086 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748089 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748092 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748095 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748097 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748100 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748103 5055 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748106 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748108 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748111 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748114 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748116 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748119 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748121 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:12.759886 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748124 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748126 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748129 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748132 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748135 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748137 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748140 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748143 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748146 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748148 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748151 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748153 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748156 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748159 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748161 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748164 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748166 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748169 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748172 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:12.760544 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748174 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748177 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748179 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748183 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748186 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748191 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748194 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748196 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748199 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748201 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748204 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748207 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748209 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748212 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748214 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748217 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748219 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748222 5055 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748224 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:12.761147 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748227 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748229 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748232 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748234 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748237 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748239 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748242 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748244 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748247 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748249 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748252 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748254 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748257 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748260 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748263 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748266 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748268 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748271 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748274 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:12.761818 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748276 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748279 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748281 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748284 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748286 5055 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748289 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748291 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748294 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748296 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748299 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748301 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748303 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748306 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748309 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748311 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748313 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748316 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748318 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748321 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748342 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:12.762308 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748345 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748347 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748350 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.748377 5055 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748507 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748512 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748514 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748517 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748520 5055 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748523 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748525 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748528 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748530 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748533 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748536 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748538 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:12.762796 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748541 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748543 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748545 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748548 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748550 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748553 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748555 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748558 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748560 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748563 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748565 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748568 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748570 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748573 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748576 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748579 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748581 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748584 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748586 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748589 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:12.763177 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748591 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748594 5055 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748596 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748599 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748601 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748604 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748607 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748609 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748612 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748614 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748617 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748620 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748622 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748624 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748627 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748629 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748632 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748634 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748637 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:12.763751 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748639 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748641 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748646 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748649 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748652 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748655 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748657 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748660 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748662 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748665 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748668 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748670 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748672 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748675 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748678 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748680 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748683 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748685 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748688 5055 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:12.764222 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748691 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748693 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748696 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748698 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748701 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748703 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748706 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748709 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748711 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748714 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748716 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748718 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748721 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748724 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748727 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748731 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748734 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748737 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748740 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:12.764706 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748743 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748745 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748748 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748752 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748756 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748759 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748762 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748765 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748769 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748771 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748774 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748776 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748779 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748781 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748784 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748787 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:12.765311 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748790 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.748811 5055 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748844 5055 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748850 5055 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748852 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748855 5055 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748858 5055 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748860 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748863 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748866 5055 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748868 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748871 5055 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748873 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748879 5055 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748881 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748884 5055 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:12.765812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748887 5055 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748889 5055 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748892 5055 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748895 5055 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748897 5055 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748900 5055 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748902 5055 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748905 5055 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748907 5055 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748910 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748913 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748915 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748918 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748921 5055 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748926 5055 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748929 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748932 5055 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748934 5055 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748937 5055 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:12.766199 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748940 5055 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748942 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748945 5055 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748948 5055 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748952 5055 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748954 5055 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748957 5055 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748959 5055 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748962 5055 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748964 5055 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748968 5055 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748971 5055 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748974 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748976 5055 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748979 5055 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748981 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748984 5055 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748987 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748990 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:12.766669 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748992 5055 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748995 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.748997 5055 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749000 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749002 5055 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749005 5055 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749008 5055 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749010 5055 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749013 5055 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749015 5055 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749019 5055 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749022 5055 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749024 5055 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749027 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749029 5055 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749032 5055 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749034 5055 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749037 5055 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749040 5055 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749042 5055 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749045 5055 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:12.767376 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749047 5055 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749050 5055 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749053 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749056 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749059 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749061 5055 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749064 5055 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749066 5055 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749069 5055 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749071 5055 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749074 5055 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749076 5055 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749079 5055 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749082 5055 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749084 5055 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749096 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749099 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749102 5055 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749104 5055 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:12.768117 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749107 5055 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749110 5055 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749112 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749115 5055 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749118 5055 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749121 5055 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749123 5055 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749126 5055 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749130 5055 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749132 5055 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749135 5055 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749138 5055 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749140 5055 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:12.749143 5055 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.749192 5055 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.750161 5055 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.750211 5055 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.750400 5055 server.go:1014] "Starting client certificate rotation" Oct 30 01:17:12.768919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.750532 5055 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:17:12.769539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.750574 5055 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:11:31 +0000 UTC" deadline="2028-08-09 08:06:45.40296364 +0000 UTC" Oct 30 01:17:12.769539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.750611 5055 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15582h49m32.652354964s" Oct 30 01:17:12.769539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.751400 5055 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:17:12.769539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.753017 5055 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:17:12.769539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.754169 5055 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:17:12.769539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.755916 5055 log.go:25] "Validated CRI v1 image API" Oct 30 01:17:12.769539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.756645 5055 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:17:12.769539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.758661 5055 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d0c94ca6-26bc-4b67-beab-cb2f8f648b10:/dev/nvme0n1p4 edd7a548-da53-49cb-adf7-188d93ff64a4:/dev/nvme0n1p3] Oct 30 01:17:12.769927 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.758676 5055 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/015b60aeb8a694313ddfb327651057a6e9449ec69363a98c0b78412de46af90c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/015b60aeb8a694313ddfb327651057a6e9449ec69363a98c0b78412de46af90c/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16dae0468c0af28ebfa64b54be4527d564599feec177245e244bfb390916d6c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16dae0468c0af28ebfa64b54be4527d564599feec177245e244bfb390916d6c1/userdata/shm major:0 minor:356 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1706b219852576de48023ea1f2304cbae21c23bccbabd5de5f54a3168faf15a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1706b219852576de48023ea1f2304cbae21c23bccbabd5de5f54a3168faf15a4/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17cb1ef5e6ccb32fd7a4049f9d2901ee7acde9ff5e2b8100424aec55d7637def/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17cb1ef5e6ccb32fd7a4049f9d2901ee7acde9ff5e2b8100424aec55d7637def/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1fbeea87e508fddc4b3b851b7144df573860432e5448d5bc8341f7d01362aa59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1fbeea87e508fddc4b3b851b7144df573860432e5448d5bc8341f7d01362aa59/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26a4ee28be21d0a49084db02816636f8b892cb1441c9cd007faaedbb29923b4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26a4ee28be21d0a49084db02816636f8b892cb1441c9cd007faaedbb29923b4a/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38368dbe1ec9a310be611eb35869a106c185febe1e2f4d4f9ca7eb5375bbf6e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38368dbe1ec9a310be611eb35869a106c185febe1e2f4d4f9ca7eb5375bbf6e9/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49a14e50a6ab971df021c9e1e711e0da45528de7acef663ee026eadb4d2d11a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49a14e50a6ab971df021c9e1e711e0da45528de7acef663ee026eadb4d2d11a8/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5586bcc51fe3d1f925e6091add497847856a02c5959decadb4c2a0144982ba6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5586bcc51fe3d1f925e6091add497847856a02c5959decadb4c2a0144982ba6e/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/604b48f2a6bf900a68d28b8a4f655902823a2c35093a4b53e5392cf9d0abfe90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/604b48f2a6bf900a68d28b8a4f655902823a2c35093a4b53e5392cf9d0abfe90/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/659e8018552c405d80f18629929fddbb026361e9a951ccf95a172f93321cd319/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/659e8018552c405d80f18629929fddbb026361e9a951ccf95a172f93321cd319/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b3ff3a129302e0236657cddceb47cb1094f1e633f34692359e065d77cd2a394/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b3ff3a129302e0236657cddceb47cb1094f1e633f34692359e065d77cd2a394/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e79aac7069a84bc8f2b06cbd90ffb81ac70e134d00bf0bf268bd2b44b4b7b5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e79aac7069a84bc8f2b06cbd90ffb81ac70e134d00bf0bf268bd2b44b4b7b5d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/967743bc57a5fffc01f1ec8c160c9dba25d32d19af6bfa5ac75eff37ceb93c6b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/967743bc57a5fffc01f1ec8c160c9dba25d32d19af6bfa5ac75eff37ceb93c6b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0da20c9274c1300ff32558da7553589ed351a415146668daef151225ec05b13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0da20c9274c1300ff32558da7553589ed351a415146668daef151225ec05b13/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3613ef44430a444c8560394f44d0ef7210b83d832738f1907fc15b6d90c131b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3613ef44430a444c8560394f44d0ef7210b83d832738f1907fc15b6d90c131b/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b9c6b81d1b651e648a6f8dbe14cccc2c4e2172dd5f3ecc71e9a0deb4fc2a72a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b9c6b81d1b651e648a6f8dbe14cccc2c4e2172dd5f3ecc71e9a0deb4fc2a72a8/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc10f8646379b177bae48b25a694c1ed7902e21be6b60df1c2af0a3f5651a57a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc10f8646379b177bae48b25a694c1ed7902e21be6b60df1c2af0a3f5651a57a/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd2f1bbed8bd0f04e30ec784b0bfcab605d5f52815145ddab8441674fabcc6ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd2f1bbed8bd0f04e30ec784b0bfcab605d5f52815145ddab8441674fabcc6ca/userdata/shm major:0 minor:343 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c608864e5122f9ff1fe5c5d9696eabf37d485e59a24c65682c6d99bc62ec7be2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c608864e5122f9ff1fe5c5d9696eabf37d485e59a24c65682c6d99bc62ec7be2/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cdd603789b0b0581f00e91ebf74be898007d44ba6a48597712bde59946488b93/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cdd603789b0b0581f00e91ebf74be898007d44ba6a48597712bde59946488b93/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d5d350864092bebeec0e23ee5d30d9483b06aabba3a025eae44cad5a0d3264b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d5d350864092bebeec0e23ee5d30d9483b06aabba3a025eae44cad5a0d3264b3/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd4f7bdb4814f7c89507166194d8182512d9eae411e8af641cae73ec672b1274/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd4f7bdb4814f7c89507166194d8182512d9eae411e8af641cae73ec672b1274/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e22fe9cb499d4fedab46c2f49f47021cc3c5614e5e0be3322b6117465b462b08/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e22fe9cb499d4fedab46c2f49f47021cc3c5614e5e0be3322b6117465b462b08/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e5e9d78a986a1f47384efa0f900544bb1293a10a69b01eb62f365e40887c82e6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e5e9d78a986a1f47384efa0f900544bb1293a10a69b01eb62f365e40887c82e6/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fad6384529a2299251911a6192babdeea6fe6065a4c8b63c516bffc1d9a495d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fad6384529a2299251911a6192babdeea6fe6065a4c8b63c516bffc1d9a495d7/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/083c7373-177e-46cb-99c1-f9509121668a/volumes/kubernetes.io~projected/kube-api-access-8nzwm:{mountpoint:/var/lib/kubelet/pods/083c7373-177e-46cb-99c1-f9509121668a/volumes/kubernetes.io~projected/kube-api-access-8nzwm major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/083c7373-177e-46cb-99c1-f9509121668a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/083c7373-177e-46cb-99c1-f9509121668a/volumes/kubernetes.io~secret/serving-cert major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3/volumes/kubernetes.io~projected/kube-api-access-2529z:{mountpoint:/var/lib/kubelet/pods/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3/volumes/kubernetes.io~projected/kube-api-access-2529z major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/184de02f-0273-408d-9ee5-265a0bb278a4/volumes/kubernetes.io~projected/kube-api-access-mf4dt:{mountpoint:/var/lib/kubelet/pods/184de02f-0273-408d-9ee5-265a0bb278a4/volumes/kubernetes.io~projected/kube-api-access-mf4dt major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5/volumes/kubernetes.io~projected/kube-api-access-8pzfs:{mountpoint:/var/lib/kubelet/pods/1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5/volumes/kubernetes.io~projected/kube-api-access-8pzfs major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e24a38d-9f35-40bf-bb90-57fee63d8ac0/volumes/kubernetes.io~projected/kube-api-access-bzmb8:{mountpoint:/var/lib/kubelet/pods/1e24a38d-9f35-40bf-bb90-57fee63d8ac0/volumes/kubernetes.io~projected/kube-api-access-bzmb8 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/335e4ad1-7154-4e2c-a170-d52aee2df045/volumes/kubernetes.io~projected/kube-api-access-dqbqr:{mountpoint:/var/lib/kubelet/pods/335e4ad1-7154-4e2c-a170-d52aee2df045/volumes/kubernetes.io~projected/kube-api-access-dqbqr major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3db07759-c867-4de9-84c7-24d903510dca/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/3db07759-c867-4de9-84c7-24d903510dca/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3db07759-c867-4de9-84c7-24d903510dca/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/3db07759-c867-4de9-84c7-24d903510dca/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3db07759-c867-4de9-84c7-24d903510dca/volumes/kubernetes.io~projected/kube-api-access-29vzs:{mountpoint:/var/lib/kubelet/pods/3db07759-c867-4de9-84c7-24d903510dca/volumes/kubernetes.io~projected/kube-api-access-29vzs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4588e6fe-4b1e-4e62-8528-7411e598b444/volumes/kubernetes.io~projected/kube-api-access-9jb47:{mountpoint:/var/lib/kubelet/pods/4588e6fe-4b1e-4e62-8528-7411e598b444/volumes/kubernetes.io~projected/kube-api-access-9jb47 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/495d54bf-6c21-46ab-badd-8ece131d6618/volumes/kubernetes.io~projected/kube-api-access-sx866:{mountpoint:/var/lib/kubelet/pods/495d54bf-6c21-46ab-badd-8ece131d6618/volumes/kubernetes.io~projected/kube-api-access-sx866 major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4a8853b4-1b14-4b16-81d8-007e7827d8dd/volumes/kubernetes.io~projected/kube-api-access-lwp4f:{mountpoint:/var/lib/kubelet/pods/4a8853b4-1b14-4b16-81d8-007e7827d8dd/volumes/kubernetes.io~projected/kube-api-access-lwp4f major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17/volumes/kubernetes.io~projected/kube-api-access-9p5mm:{mountpoint:/var/lib/kubelet/pods/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17/volumes/kubernetes.io~projected/kube-api-access-9p5mm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56fd7c23-5b71-424f-87e6-91f34918efe8/volumes/kubernetes.io~projected/kube-api-access-fjxrh:{mountpoint:/var/lib/kubelet/pods/56fd7c23-5b71-424f-87e6-91f34918efe8/volumes/kubernetes.io~projected/kube-api-access-fjxrh major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591/volumes/kubernetes.io~projected/kube-api-access-7vgmg:{mountpoint:/var/lib/kubelet/pods/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591/volumes/kubernetes.io~projected/kube-api-access-7vgmg major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/628af330-6652-4df0-8275-9f691e2c8eb3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/628af330-6652-4df0-8275-9f691e2c8eb3/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~projected/kube-api-access-46q9r:{mountpoint:/var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~projected/kube-api-access-46q9r major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/712bffb7-97f2-422a-ba7b-cce9eafccba4/volumes/kubernetes.io~projected/kube-api-access-6khjk:{mountpoint:/var/lib/kubelet/pods/712bffb7-97f2-422a-ba7b-cce9eafccba4/volumes/kubernetes.io~projected/kube-api-access-6khjk major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/712bffb7-97f2-422a-ba7b-cce9eafccba4/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/712bffb7-97f2-422a-ba7b-cce9eafccba4/volumes/kubernetes.io~secret/serving-cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~projected/kube-api-access-2tqtw:{mountpoint:/var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~projected/kube-api-access-2tqtw major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8855c2e7-9182-4ae6-b96a-50baa69d9a67/volumes/kubernetes.io~projected/kube-api-access-z2cf9:{mountpoint:/var/lib/kubelet/pods/8855c2e7-9182-4ae6-b96a-50baa69d9a67/volumes/kubernetes.io~projected/kube-api-access-z2cf9 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8855c2e7-9182-4ae6-b96a-50baa69d9a67/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8855c2e7-9182-4ae6-b96a-50baa69d9a67/volumes/kubernetes.io~secret/cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bc436c0-2b1b-477e-81aa-af1a1eff9779/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8bc436c0-2b1b-477e-81aa-af1a1eff9779/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97b353f9-eeeb-4f6a-bdb7-204450f76167/volumes/kubernetes.io~projected/kube-api-access-bzcm2:{mountpoint:/var/lib/kubelet/pods/97b353f9-eeeb-4f6a-bdb7-204450f76167/volumes/kubernetes.io~projected/kube-api-access-bzcm2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db/volumes/kubernetes.io~projected/kube-api-access-vcv6t:{mountpoint:/var/lib/kubelet/pods/98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db/volumes/kubernetes.io~projected/kube-api-access-vcv6t major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f4577e2-2c6f-4faf-a425-4ce3e00cec84/volumes/kubernetes.io~projected/kube-api-access-h7cmp:{mountpoint:/var/lib/kubelet/pods/9f4577e2-2c6f-4faf-a425-4ce3e00cec84/volumes/kubernetes.io~projected/kube-api-access-h7cmp major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f4577e2-2c6f-4faf-a425-4ce3e00cec84/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/9f4577e2-2c6f-4faf-a425-4ce3e00cec84/volumes/kubernetes.io~secret/signing-key major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a310cb37-d20b-4b25-9796-c5f56c44adac/volumes/kubernetes.io~projected/kube-api-access-jblp7:{mountpoint:/var/lib/kubelet/pods/a310cb37-d20b-4b25-9796-c5f56c44adac/volumes/kubernetes.io~projected/kube-api-access-jblp7 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a310cb37-d20b-4b25-9796-c5f56c44adac/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/a310cb37-d20b-4b25-9796-c5f56c44adac/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a310cb37-d20b-4b25-9796-c5f56c44adac/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/a310cb37-d20b-4b25-9796-c5f56c44adac/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b47f8e4a-5060-47bf-8bec-722ccff0fa51/volumes/kubernetes.io~projected/kube-api-access-hrqvr:{mountpoint:/var/lib/kubelet/pods/b47f8e4a-5060-47bf-8bec-722ccff0fa51/volumes/kubernetes.io~projected/kube-api-access-hrqvr major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4e79600-74c8-4e22-8fce-fbd02aca1e38/volumes/kubernetes.io~projected/kube-api-access-6k4bh:{mountpoint:/var/lib/kubelet/pods/b4e79600-74c8-4e22-8fce-fbd02aca1e38/volumes/kubernetes.io~projected/kube-api-access-6k4bh major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/volumes/kubernetes.io~projected/kube-api-access-f8q47:{mountpoint:/var/lib/kubelet/pods/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/volumes/kubernetes.io~projected/kube-api-access-f8q47 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/volumes/kubernetes.io~secret/serving-cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb940654-4320-4340-a2d5-9450bc68f40f/volumes/kubernetes.io~projected/kube-api-access-6jwhg:{mountpoint:/var/lib/kubelet/pods/bb940654-4320-4340-a2d5-9450bc68f40f/volumes/kubernetes.io~projected/kube-api-access-6jwhg major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bb940654-4320-4340-a2d5-9450bc68f40f/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bb940654-4320-4340-a2d5-9450bc68f40f/volumes/kubernetes.io~secret/serving-cert major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9801105-8f8a-45d0-b2c1-23468a212347/volumes/kubernetes.io~projected/kube-api-access-trgqt:{mountpoint:/var/lib/kubelet/pods/c9801105-8f8a-45d0-b2c1-23468a212347/volumes/kubernetes.io~projected/kube-api-access-trgqt major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c9b752e3-e38a-4d0f-babd-7922d6a2e872/volumes/kubernetes.io~projected/kube-api-access-gnwz2:{mountpoint:/var/lib/kubelet/pods/c9b752e3-e38a-4d0f-babd-7922d6a2e872/volumes/kubernetes.io~projected/kube-api-access-gnwz2 major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc0b4765-071a-4af8-88e6-700abd1d8f83/volumes/kubernetes.io~projected/kube-api-access-k4d5n:{mountpoint:/var/lib/kubelet/pods/cc0b4765-071a-4af8-88e6-700abd1d8f83/volumes/kubernetes.io~projected/kube-api-access-k4d5n major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d8c54d89-eea4-45ee-80e5-fd0aba53e274/volumes/kubernetes.io~projected/kube-api-access-nwd46:{mountpoint:/var/lib/kubelet/pods/d8c54d89-eea4-45ee-80e5-fd0aba53e274/volumes/kubernetes.io~projected/kube-api-access-nwd46 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbf09128-a8c1-4d32-958b-cbd01b892097/volumes/kubernetes.io~projected/kube-api-access-qq8g2:{mountpoint:/var/lib/kubelet/pods/dbf09128-a8c1-4d32-958b-cbd01b892097/volumes/kubernetes.io~projected/kube-api-access-qq8g2 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f207f3a7-cf1c-4acc-8847-267e7c89e31f/volumes/kubernetes.io~projected/kube-api-access-25s9d:{mountpoint:/var/lib/kubelet/pods/f207f3a7-cf1c-4acc-8847-267e7c89e31f/volumes/kubernetes.io~projected/kube-api-access-25s9d major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c16f56c0c5e601ba4590c3bef787e63e6a4634b11508eda42eed609dbee09c2a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/8f69f4817f89f0cf79cd1d5ab7922c18c3f22ada9bd81d9e06b07090be7e0257/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/b429e08935cde7358dc74f155efc1f8da722f83daf3aae00b635e00b17f00228/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/cd968528c5e8707e413ff8e662b4b0e6b62d60512c0e5fa72e2737893afced30/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ed3df52dc5c71628c933fcd52897a674cd8eb5684dc8cda36d2659a5d8e6386a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/e2183b3c79248b09f3c78f0932017ad6a46c9accf00bb84f79bc3216f881c809/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/7bb4c84695f2448cc10f7e40368c1340074e8a3c589114e86364b3ae3da82ab1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/90d9c0ccca7492d0ec8948b2fc0fc6704c2d89cdc0341484007fcc2bc8319adb/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/bcbd970c7e556772cb1c5e4da6ca6e5b00466298a5e938e31483f2455e96fc54/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/56ef1ac33b7826caba167fab414285b9f244900a49764746cdec6df5fe6405af/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/31de64a40cd4a9a8befd4e299bf0bcf4b92d04751cc08733c904c968ceebc8c3/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/3b6b040caebe58a9c7942c973a2c405e8edd18901447145171c6c4c9352a8ad2/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9c27addb4c57ad2fa7f4b1459472cbef94cb2fc7f2eedd676a97c90ea70813d9/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/25efc5af178ec34d5dc618e322fdeb72a107cd7531c9db1393728486728858c0/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/78478f65b176c43fa82a0b79939592db4d3f58ade00e32524a509f948722294b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-164:{mountpoint:/var/lib/containers/storage/overlay/3fc0b7cda3691ebe1d590adc16cfcbb55c64b67a3371c19d2a182d6149b38332/merged major:0 minor:164 fsType:overlay blockSize:0} overlay_0-168:{mountpoint:/var/lib/containers/storage/overlay/faf03a21ef259d3cca4709a0d06aff439a02215d3071a2104e9416a067225d49/merged major:0 minor:168 fsType:overlay blockSize:0} overlay_0-170:{mountpoint:/var/lib/containers/storage/overlay/6b40ad0d86b9a9724de1920c12016c66e184d1cc30acc449add64d6f533aac12/merged major:0 minor:170 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/22d058f7fd7d47952eb5a10fd2ba67b0cf1cc1c69e8767d90e6c557388732342/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/476d1d23fcf782960a508dde266313dd7517f1cdd5c904aaa13e681f5ca1ddc3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/4ea1cba5c7af33e8fcdc330926b8143caa2811f668c0b45db3803b663f87c762/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/e80bec9392b2400f14ede1cdc458bb8e5fbdd079f49ca063d66d40b0b0695c25/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/808b92f68228d394368dd0d5cd312fdbe4b008e131695dd78bd0d3770838f207/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/21bc83255a74c53ba02c7f33f13e7e0f6b796c74f2274480218e8a0cd1a3ef2a/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/326f044fce3279595d0054689470d638f2e69aeca75790185e33c08c4fcd2b4a/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/4c7825813d26b1dffe467eac0887d3d6623d903aa7b368264108247de0aa0091/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/1fc1ee0cca580d1096873da1960d3f2511c0abe108260c698b8f21b549a9e22f/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/4e83b4a12c8a170f148f28742dc13368b3391858125a6cb7ebb992c6f7f54941/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/150c2b6c8ef33c6aff7efa65af7d169c3335585838d3fadfd6cb4e10338a04b0/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/9d767823146c11cc105ba13d0de7bad115523bb55fa42d6a80578eef0dc81a67/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/bb242c61d22eba84dac53b13a8e032df4bf39d26d283a917c6d88ed4abe72e6f/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/5bc4613bb60d69abf87577c038274b2c1121ae85c40337de7924160a04b5c72a/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/0efc0f4f36752d1493c5da8500dd9c0989b7af588dfc87bc906f2bd92f875704/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/fab04def97357f9c60f57fe4e9cd0db2cda4e2f777cae40147d40ff3353cc76b/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/86aada884af0fc99cc622f22fa4a18d9cf9274fcad391630f9aa6828b77086ca/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/97c1831ce7ffbeae6a0284c7d29ad2dd1922aaa29c6738f9043b15c9e8678b1b/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/94f80725207ed587037a860e066c4f60354091fe8b3b7022788850d066ac4aad/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/283653e61a82673f04974490cc26b6d8dd96795164032299298cee1195a4751e/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/4d36e391c6f9196f406541b0474bdea393afbe5e9d1e80d5fa56437eec81b38e/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/5e93d99943195647100c99071c4dddcc792974237e6d428056bd4d320fd69c9c/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/2ec8969c8915545a24aa549139d743d48b6c376002c340c6ded218c19a3f2720/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/258757da08da2ce8ae083ddf8eee67f59a5a52258b839f17c19bfcb2fbfe3f44/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/06ecb150883f488adafebf9b4a2b73e99732e357d977e9a9d8bb90c338bb57d9/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/f98ba93980b016fd69a7775668da3d4b54206c0971b812cec4d94b9aabb10548/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/e463b909f2e79ea7382e18c071aa8ea3c84987c0f6e3f5f3f4aefbb5cdc695ee/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/ea2f1d88b3b039393cb7ef625ccd629b0b6f5db25570b0675d92129970f2da6d/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-358:{mountpoint:/var/lib/containers/storage/overlay/b9753e19b56cbc7fc88a1cbd6e3f972eb8df01733594d57496a4768afc68b6cd/merged major:0 minor:358 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/7492bad049ea3ead7785c92745fe9441ed4e54112e2120ea8a125e332bbf8d6d/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/60e576536fbc7adeb077db783134e2718f7504a7a76483247a71354d97560790/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/8496cbfd20320322133d8d187551d14e918979d9fcb821019c5bd46bc822d6fd/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/8ea013dbf657ab96cf94f44561101d6670c1edb6e33334824f3b5f8185a78c6a/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/83fe220eccafe314573931953b4dcf55eba937d16cbfcbf729d69715a41292e2/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/f3916bfa0a55321698036d26ae67e6589273cf43d5adddafeaf9b3803be7538b/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/5ef22b4711fa8028798b28755ebd1cbaaf036002468d08632dd6f4a825c67235/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/0a87fbf09981ed0b2455fd049270aa949de77bbad46ef7c1219031df9c207437/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-75:{mountpoint:/var/lib/containers/storage/overlay/b57379944a209244072a7e14413bdc59eee0691856a5b03bd28f0b55bb50fce8/merged major:0 minor:75 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/9b1d4f780c085996d3d74cccfaa0007ad0d62b1c8854bacbca96a70de1df36ba/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/1eab78bd4fa5271cc9b6dcdf57b14f36bf86d03d81eb7dcf261eea42bd304859/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/e84e4b60ace6e8c135331128fa755061d78a663bd1f450a0d52ea98b434aa636/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/6fd17db9305469d099f0594881794581d626d3403f6848d56b47c4ed35d83c12/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6d92d15eb1fa829cd26a034a919eb6b1c2a9c1194a4f3c695648fad2fa0ccc90/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/76c8739ad7c77cafe4c84b2506ff04156800a1294d0d0f77237df1acf33ad59e/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.767138 5055 manager.go:217] Machine: {Timestamp:2026-10-30 01:17:12.766674597 +0000 UTC m=+0.087631631 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099723 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e83693b95fc7e2d31cdc46e86d507 SystemUUID:ec2e8369-3b95-fc7e-2d31-cdc46e86d507 BootID:1ada4c04-365c-4a45-b0ba-61eba918e8e4 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a310cb37-d20b-4b25-9796-c5f56c44adac/volumes/kubernetes.io~projected/kube-api-access-jblp7 DeviceMajor:0 DeviceMinor:214 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-358 DeviceMajor:0 DeviceMinor:358 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/015b60aeb8a694313ddfb327651057a6e9449ec69363a98c0b78412de46af90c/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc0b4765-071a-4af8-88e6-700abd1d8f83/volumes/kubernetes.io~projected/kube-api-access-k4d5n DeviceMajor:0 DeviceMinor:339 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5/volumes/kubernetes.io~projected/kube-api-access-8pzfs DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e5e9d78a986a1f47384efa0f900544bb1293a10a69b01eb62f365e40887c82e6/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9f4577e2-2c6f-4faf-a425-4ce3e00cec84/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:352 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a310cb37-d20b-4b25-9796-c5f56c44adac/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c9b752e3-e38a-4d0f-babd-7922d6a2e872/volumes/kubernetes.io~projected/kube-api-access-gnwz2 DeviceMajor:0 DeviceMinor:212 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26a4ee28be21d0a49084db02816636f8b892cb1441c9cd007faaedbb29923b4a/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd4f7bdb4814f7c89507166194d8182512d9eae411e8af641cae73ec672b1274/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16dae0468c0af28ebfa64b54be4527d564599feec177245e244bfb390916d6c1/userdata/shm DeviceMajor:0 DeviceMinor:356 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cdd603789b0b0581f00e91ebf74be898007d44ba6a48597712bde59946488b93/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3db07759-c867-4de9-84c7-24d903510dca/volumes/kubernetes.io~projected/kube-api-access-29vzs DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/97b353f9-eeeb-4f6a-bdb7-204450f76167/volumes/kubernetes.io~projected/kube-api-access-bzcm2 DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/335e4ad1-7154-4e2c-a170-d52aee2df045/volumes/kubernetes.io~projected/kube-api-access-dqbqr DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db/volumes/kubernetes.io~projected/kube-api-access-vcv6t DeviceMajor:0 DeviceMinor:226 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dbf09128-a8c1-4d32-958b-cbd01b892097/volumes/kubernetes.io~projected/kube-api-access-qq8g2 DeviceMajor:0 DeviceMinor:293 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1fbeea87e508fddc4b3b851b7144df573860432e5448d5bc8341f7d01362aa59/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/184de02f-0273-408d-9ee5-265a0bb278a4/volumes/kubernetes.io~projected/kube-api-access-mf4dt DeviceMajor:0 DeviceMinor:310 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~projected/kube-api-access-2tqtw DeviceMajor:0 DeviceMinor:215 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/083c7373-177e-46cb-99c1-f9509121668a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:219 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/495d54bf-6c21-46ab-badd-8ece131d6618/volumes/kubernetes.io~projected/kube-api-access-sx866 DeviceMajor:0 DeviceMinor:220 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/604b48f2a6bf900a68d28b8a4f655902823a2c35093a4b53e5392cf9d0abfe90/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3/volumes/kubernetes.io~projected/kube-api-access-2529z DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8855c2e7-9182-4ae6-b96a-50baa69d9a67/volumes/kubernetes.io~projected/kube-api-access-z2cf9 DeviceMajor:0 DeviceMinor:222 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3613ef44430a444c8560394f44d0ef7210b83d832738f1907fc15b6d90c131b/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fad6384529a2299251911a6192babdeea6fe6065a4c8b63c516bffc1d9a495d7/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8855c2e7-9182-4ae6-b96a-50baa69d9a67/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:217 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/659e8018552c405d80f18629929fddbb026361e9a951ccf95a172f93321cd319/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1706b219852576de48023ea1f2304cbae21c23bccbabd5de5f54a3168faf15a4/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9f4577e2-2c6f-4faf-a425-4ce3e00cec84/volumes/kubernetes.io~projected/kube-api-access-h7cmp DeviceMajor:0 DeviceMinor:353 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38368dbe1ec9a310be611eb35869a106c185febe1e2f4d4f9ca7eb5375bbf6e9/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:209 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3db07759-c867-4de9-84c7-24d903510dca/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56fd7c23-5b71-424f-87e6-91f34918efe8/volumes/kubernetes.io~projected/kube-api-access-fjxrh DeviceMajor:0 DeviceMinor:336 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/967743bc57a5fffc01f1ec8c160c9dba25d32d19af6bfa5ac75eff37ceb93c6b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/712bffb7-97f2-422a-ba7b-cce9eafccba4/volumes/kubernetes.io~projected/kube-api-access-6khjk DeviceMajor:0 DeviceMinor:207 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:211 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591/volumes/kubernetes.io~projected/kube-api-access-7vgmg DeviceMajor:0 DeviceMinor:334 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a310cb37-d20b-4b25-9796-c5f56c44adac/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c608864e5122f9ff1fe5c5d9696eabf37d485e59a24c65682c6d99bc62ec7be2/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bb940654-4320-4340-a2d5-9450bc68f40f/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:218 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b47f8e4a-5060-47bf-8bec-722ccff0fa51/volumes/kubernetes.io~projected/kube-api-access-hrqvr DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/628af330-6652-4df0-8275-9f691e2c8eb3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/volumes/kubernetes.io~projected/kube-api-access-f8q47 DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b9c6b81d1b651e648a6f8dbe14cccc2c4e2172dd5f3ecc71e9a0deb4fc2a72a8/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e79aac7069a84bc8f2b06cbd90ffb81ac70e134d00bf0bf268bd2b44b4b7b5d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/083c7373-177e-46cb-99c1-f9509121668a/volumes/kubernetes.io~projected/kube-api-access-8nzwm DeviceMajor:0 DeviceMinor:224 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bc436c0-2b1b-477e-81aa-af1a1eff9779/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~projected/kube-api-access-46q9r DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/712bffb7-97f2-422a-ba7b-cce9eafccba4/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:205 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b3ff3a129302e0236657cddceb47cb1094f1e633f34692359e065d77cd2a394/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f207f3a7-cf1c-4acc-8847-267e7c89e31f/volumes/kubernetes.io~projected/kube-api-access-25s9d DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d5d350864092bebeec0e23ee5d30d9483b06aabba3a025eae44cad5a0d3264b3/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0da20c9274c1300ff32558da7553589ed351a415146668daef151225ec05b13/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd2f1bbed8bd0f04e30ec784b0bfcab605d5f52815145ddab8441674fabcc6ca/userdata/shm DeviceMajor:0 DeviceMinor:343 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-164 DeviceMajor:0 DeviceMinor:164 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17cb1ef5e6ccb32fd7a4049f9d2901ee7acde9ff5e2b8100424aec55d7637def/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc10f8646379b177bae48b25a694c1ed7902e21be6b60df1c2af0a3f5651a57a/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/c9801105-8f8a-45d0-b2c1-23468a212347/volumes/kubernetes.io~projected/kube-api-access-trgqt DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d8c54d89-eea4-45ee-80e5-fd0aba53e274/volumes/kubernetes.io~projected/kube-api-access-nwd46 DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-168 DeviceMajor:0 DeviceMinor:168 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bb940654-4320-4340-a2d5-9450bc68f40f/volumes/kubernetes.io~projected/kube-api-access-6jwhg DeviceMajor:0 DeviceMinor:221 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-170 DeviceMajor:0 DeviceMinor:170 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e22fe9cb499d4fedab46c2f49f47021cc3c5614e5e0be3322b6117465b462b08/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49a14e50a6ab971df021c9e1e711e0da45528de7acef663ee026eadb4d2d11a8/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3db07759-c867-4de9-84c7-24d903510dca/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4a8853b4-1b14-4b16-81d8-007e7827d8dd/volumes/kubernetes.io~projected/kube-api-access-lwp4f DeviceMajor:0 DeviceMinor:216 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5586bcc51fe3d1f925e6091add497847856a02c5959decadb4c2a0144982ba6e/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17/volumes/kubernetes.io~projected/kube-api-access-9p5mm DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-75 DeviceMajor:0 DeviceMinor:75 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4e79600-74c8-4e22-8fce-fbd02aca1e38/volumes/kubernetes.io~projected/kube-api-access-6k4bh DeviceMajor:0 DeviceMinor:210 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e24a38d-9f35-40bf-bb90-57fee63d8ac0/volumes/kubernetes.io~projected/kube-api-access-bzmb8 DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4588e6fe-4b1e-4e62-8528-7411e598b444/volumes/kubernetes.io~projected/kube-api-access-9jb47 DeviceMajor:0 DeviceMinor:227 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:16dae0468c0af28 MacAddress:a2:f0:23:35:7f:f6 Speed:10000 Mtu:8901} {Name:1706b219852576d MacAddress:32:6e:fe:59:26:80 Speed:10000 Mtu:8901} {Name:17cb1ef5e6ccb32 MacAddress:5e:b9:15:50:cb:45 Speed:10000 Mtu:8901} {Name:26a4ee28be21d0a MacAddress:52:f0:77:b3:19:73 Speed:10000 Mtu:8901} {Name:49a14e50a6ab971 MacAddress:42:c4:03:b7:9c:02 Speed:10000 Mtu:8901} {Name:5586bcc51fe3d1f MacAddress:1a:75:b9:a6:d6:af Speed:10000 Mtu:8901} {Name:659e8018552c405 MacAddress:d2:3f:b6:02:58:6f Speed:10000 Mtu:8901} {Name:6b3ff3a129302e0 MacAddress:e6:13:48:7f:7a:26 Speed:10000 Mtu:8901} {Name:a0da20c9274c130 MacAddress:9a:0c:17:c2:e0:0e Speed:10000 Mtu:8901} {Name:b3613ef44430a44 MacAddress:56:a7:9e:06:7a:24 Speed:10000 Mtu:8901} {Name:b9c6b81d1b651e6 MacAddress:42:88:96:99:6a:50 Speed:10000 Mtu:8901} {Name:bd2f1bbed8bd0f0 MacAddress:0a:c6:cb:99:b4:f8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d4:42:d9:87 Speed:0 Mtu:9001} {Name:br-int MacAddress:72:09:01:47:c5:36 Speed:0 Mtu:8901} {Name:d5d350864092beb MacAddress:fe:f1:da:34:a7:df Speed:10000 Mtu:8901} {Name:dd4f7bdb4814f7c MacAddress:2a:90:a5:b1:b8:e9 Speed:10000 Mtu:8901} {Name:e22fe9cb499d4fe MacAddress:86:4b:c2:7e:bc:18 Speed:10000 Mtu:8901} {Name:e5e9d78a986a1f4 MacAddress:ba:02:cb:95:d3:64 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d4:42:d9:87 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:96:23:bb:32:53:35 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6e:8f:ef:61:bb:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.767522 5055 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.767563 5055 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.767902 5055 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.767930 5055 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-254.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768112 5055 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768124 5055 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768142 5055 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768168 5055 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768195 5055 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768437 5055 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768571 5055 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768614 5055 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768632 5055 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768646 5055 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768659 5055 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.768670 5055 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.770212 5055 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:17:12.770318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.770229 5055 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:17:12.771463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.770852 5055 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:17:12.773446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773432 5055 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:17:12.773446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773446 5055 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:17:12.773553 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773542 5055 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 01:17:12.773728 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773718 5055 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:17:12.773768 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773747 5055 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:17:12.773839 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773830 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:17:12.773867 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773844 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:17:12.773867 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773850 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:17:12.773867 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773855 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:17:12.773867 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773861 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:17:12.773867 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773866 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:17:12.774014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773872 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:17:12.774014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773878 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:17:12.774014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773884 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:17:12.774014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773889 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:17:12.774014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773898 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:17:12.774014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773908 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:17:12.774014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773931 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:17:12.774014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.773936 5055 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:17:12.777287 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.777275 5055 server.go:1263] "Started kubelet" Oct 30 01:17:12.777391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.777364 5055 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:17:12.777451 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.777379 5055 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:17:12.777541 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.777452 5055 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:17:12.777881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.777862 5055 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:17:12.777903 ip-10-0-134-254 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:17:12.780928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.780911 5055 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:17:12.781047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.780987 5055 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:17:12.781437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.780971 5055 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:17:12.781437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.781421 5055 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:17:12.781437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.781432 5055 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:17:12.781680 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.781574 5055 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:17:12.781902 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.781788 5055 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:11:33 +0000 UTC" deadline="2028-05-03 08:26:26.894330564 +0000 UTC" Oct 30 01:17:12.781902 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.781839 5055 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13231h9m14.112494636s" Oct 30 01:17:12.783044 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:12.782923 5055 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-134-254.ec2.internal\" not found" Oct 30 01:17:12.785548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.785532 5055 factory.go:55] Registering systemd factory Oct 30 01:17:12.785720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.785657 5055 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.786910 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7082f68a-2079-426e-997f-1d3ba3997ba6" volumeName="kubernetes.io/empty-dir/7082f68a-2079-426e-997f-1d3ba3997ba6-ca-trust-extracted" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.786954 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" volumeName="kubernetes.io/projected/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-kube-api-access-f8q47" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.786970 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb940654-4320-4340-a2d5-9450bc68f40f" volumeName="kubernetes.io/configmap/bb940654-4320-4340-a2d5-9450bc68f40f-config" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.786982 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9b752e3-e38a-4d0f-babd-7922d6a2e872" volumeName="kubernetes.io/configmap/c9b752e3-e38a-4d0f-babd-7922d6a2e872-telemetry-config" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.786999 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7082f68a-2079-426e-997f-1d3ba3997ba6" volumeName="kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-installation-pull-secrets" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787013 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b47f8e4a-5060-47bf-8bec-722ccff0fa51" volumeName="kubernetes.io/projected/b47f8e4a-5060-47bf-8bec-722ccff0fa51-kube-api-access-hrqvr" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787030 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" volumeName="kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-config" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787043 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d8c54d89-eea4-45ee-80e5-fd0aba53e274" volumeName="kubernetes.io/projected/d8c54d89-eea4-45ee-80e5-fd0aba53e274-kube-api-access-nwd46" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787065 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806fd3b6-00de-4f42-8c35-e638d3835ec7" volumeName="kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-image-registry-private-configuration" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787078 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3" volumeName="kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-script-lib" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787090 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e24a38d-9f35-40bf-bb90-57fee63d8ac0" volumeName="kubernetes.io/projected/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-kube-api-access-bzmb8" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787108 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="335e4ad1-7154-4e2c-a170-d52aee2df045" volumeName="kubernetes.io/projected/335e4ad1-7154-4e2c-a170-d52aee2df045-kube-api-access-dqbqr" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787139 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="628af330-6652-4df0-8275-9f691e2c8eb3" volumeName="kubernetes.io/secret/628af330-6652-4df0-8275-9f691e2c8eb3-agent-certs" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787156 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806fd3b6-00de-4f42-8c35-e638d3835ec7" volumeName="kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-certificates" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787176 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" volumeName="kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-trusted-ca" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787192 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f207f3a7-cf1c-4acc-8847-267e7c89e31f" volumeName="kubernetes.io/projected/f207f3a7-cf1c-4acc-8847-267e7c89e31f-kube-api-access-25s9d" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787205 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083c7373-177e-46cb-99c1-f9509121668a" volumeName="kubernetes.io/projected/083c7373-177e-46cb-99c1-f9509121668a-kube-api-access-8nzwm" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787217 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" volumeName="kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-service-ca-bundle" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787233 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9801105-8f8a-45d0-b2c1-23468a212347" volumeName="kubernetes.io/configmap/c9801105-8f8a-45d0-b2c1-23468a212347-serviceca" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787244 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="495d54bf-6c21-46ab-badd-8ece131d6618" volumeName="kubernetes.io/projected/495d54bf-6c21-46ab-badd-8ece131d6618-kube-api-access-sx866" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787262 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7082f68a-2079-426e-997f-1d3ba3997ba6" volumeName="kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-bound-sa-token" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787274 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8855c2e7-9182-4ae6-b96a-50baa69d9a67" volumeName="kubernetes.io/projected/8855c2e7-9182-4ae6-b96a-50baa69d9a67-kube-api-access-z2cf9" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787286 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb940654-4320-4340-a2d5-9450bc68f40f" volumeName="kubernetes.io/secret/bb940654-4320-4340-a2d5-9450bc68f40f-serving-cert" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787302 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc0b4765-071a-4af8-88e6-700abd1d8f83" volumeName="kubernetes.io/projected/cc0b4765-071a-4af8-88e6-700abd1d8f83-kube-api-access-k4d5n" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787316 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806fd3b6-00de-4f42-8c35-e638d3835ec7" volumeName="kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-kube-api-access-2tqtw" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787358 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5" volumeName="kubernetes.io/projected/1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5-kube-api-access-8pzfs" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787371 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e24a38d-9f35-40bf-bb90-57fee63d8ac0" volumeName="kubernetes.io/configmap/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-config-volume" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787385 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17" volumeName="kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cni-binary-copy" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787408 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7082f68a-2079-426e-997f-1d3ba3997ba6" volumeName="kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-trusted-ca" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787425 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7082f68a-2079-426e-997f-1d3ba3997ba6" volumeName="kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-image-registry-private-configuration" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787436 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712bffb7-97f2-422a-ba7b-cce9eafccba4" volumeName="kubernetes.io/secret/712bffb7-97f2-422a-ba7b-cce9eafccba4-serving-cert" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787450 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db" volumeName="kubernetes.io/projected/98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db-kube-api-access-vcv6t" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787461 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083c7373-177e-46cb-99c1-f9509121668a" volumeName="kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787474 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083c7373-177e-46cb-99c1-f9509121668a" volumeName="kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-service-ca-bundle" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787490 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712bffb7-97f2-422a-ba7b-cce9eafccba4" volumeName="kubernetes.io/configmap/712bffb7-97f2-422a-ba7b-cce9eafccba4-config" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787501 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f4577e2-2c6f-4faf-a425-4ce3e00cec84" volumeName="kubernetes.io/projected/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-kube-api-access-h7cmp" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787518 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f4577e2-2c6f-4faf-a425-4ce3e00cec84" volumeName="kubernetes.io/secret/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-key" seLinuxMountContext="" Oct 30 01:17:12.787552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787528 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b47f8e4a-5060-47bf-8bec-722ccff0fa51" volumeName="kubernetes.io/empty-dir/b47f8e4a-5060-47bf-8bec-722ccff0fa51-tmp-dir" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787773 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" volumeName="kubernetes.io/secret/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-serving-cert" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787805 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bb940654-4320-4340-a2d5-9450bc68f40f" volumeName="kubernetes.io/projected/bb940654-4320-4340-a2d5-9450bc68f40f-kube-api-access-6jwhg" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787818 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41" volumeName="kubernetes.io/configmap/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-nginx-conf" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787829 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="335e4ad1-7154-4e2c-a170-d52aee2df045" volumeName="kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787841 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" volumeName="kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787852 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="712bffb7-97f2-422a-ba7b-cce9eafccba4" volumeName="kubernetes.io/projected/712bffb7-97f2-422a-ba7b-cce9eafccba4-kube-api-access-6khjk" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787863 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc436c0-2b1b-477e-81aa-af1a1eff9779" volumeName="kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787875 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e79600-74c8-4e22-8fce-fbd02aca1e38" volumeName="kubernetes.io/projected/b4e79600-74c8-4e22-8fce-fbd02aca1e38-kube-api-access-6k4bh" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787893 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9801105-8f8a-45d0-b2c1-23468a212347" volumeName="kubernetes.io/projected/c9801105-8f8a-45d0-b2c1-23468a212347-kube-api-access-trgqt" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787904 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c9b752e3-e38a-4d0f-babd-7922d6a2e872" volumeName="kubernetes.io/projected/c9b752e3-e38a-4d0f-babd-7922d6a2e872-kube-api-access-gnwz2" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787915 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3" volumeName="kubernetes.io/projected/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-kube-api-access-2529z" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787926 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3" volumeName="kubernetes.io/secret/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787938 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3db07759-c867-4de9-84c7-24d903510dca" volumeName="kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-etc-tuned" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787948 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17" volumeName="kubernetes.io/projected/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-kube-api-access-9p5mm" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787960 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7082f68a-2079-426e-997f-1d3ba3997ba6" volumeName="kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-certificates" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787970 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7082f68a-2079-426e-997f-1d3ba3997ba6" volumeName="kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-kube-api-access-46q9r" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787980 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806fd3b6-00de-4f42-8c35-e638d3835ec7" volumeName="kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-trusted-ca" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.787991 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8855c2e7-9182-4ae6-b96a-50baa69d9a67" volumeName="kubernetes.io/secret/8855c2e7-9182-4ae6-b96a-50baa69d9a67-cert" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788012 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806fd3b6-00de-4f42-8c35-e638d3835ec7" volumeName="kubernetes.io/empty-dir/806fd3b6-00de-4f42-8c35-e638d3835ec7-ca-trust-extracted" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788025 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083c7373-177e-46cb-99c1-f9509121668a" volumeName="kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-snapshots" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788037 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e24a38d-9f35-40bf-bb90-57fee63d8ac0" volumeName="kubernetes.io/empty-dir/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-tmp-dir" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788044 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="335e4ad1-7154-4e2c-a170-d52aee2df045" volumeName="kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788052 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17" volumeName="kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-daemon-config" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788059 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083c7373-177e-46cb-99c1-f9509121668a" volumeName="kubernetes.io/secret/083c7373-177e-46cb-99c1-f9509121668a-serving-cert" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788067 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184de02f-0273-408d-9ee5-265a0bb278a4" volumeName="kubernetes.io/empty-dir/184de02f-0273-408d-9ee5-265a0bb278a4-data-volume" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788074 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="184de02f-0273-408d-9ee5-265a0bb278a4" volumeName="kubernetes.io/projected/184de02f-0273-408d-9ee5-265a0bb278a4-kube-api-access-mf4dt" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788082 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3db07759-c867-4de9-84c7-24d903510dca" volumeName="kubernetes.io/projected/3db07759-c867-4de9-84c7-24d903510dca-kube-api-access-29vzs" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788089 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4a8853b4-1b14-4b16-81d8-007e7827d8dd" volumeName="kubernetes.io/projected/4a8853b4-1b14-4b16-81d8-007e7827d8dd-kube-api-access-lwp4f" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788097 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="628af330-6652-4df0-8275-9f691e2c8eb3" volumeName="kubernetes.io/configmap/628af330-6652-4df0-8275-9f691e2c8eb3-konnectivity-ca" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788104 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f207f3a7-cf1c-4acc-8847-267e7c89e31f" volumeName="kubernetes.io/configmap/f207f3a7-cf1c-4acc-8847-267e7c89e31f-iptables-alerter-script" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788120 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="083c7373-177e-46cb-99c1-f9509121668a" volumeName="kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-tmp" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788127 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="335e4ad1-7154-4e2c-a170-d52aee2df045" volumeName="kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-binary-copy" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788136 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4588e6fe-4b1e-4e62-8528-7411e598b444" volumeName="kubernetes.io/projected/4588e6fe-4b1e-4e62-8528-7411e598b444-kube-api-access-9jb47" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788143 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806fd3b6-00de-4f42-8c35-e638d3835ec7" volumeName="kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-installation-pull-secrets" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788150 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97b353f9-eeeb-4f6a-bdb7-204450f76167" volumeName="kubernetes.io/projected/97b353f9-eeeb-4f6a-bdb7-204450f76167-kube-api-access-bzcm2" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788157 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a310cb37-d20b-4b25-9796-c5f56c44adac" volumeName="kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-default-certificate" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788164 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" volumeName="kubernetes.io/empty-dir/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-archives-path" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788171 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f4577e2-2c6f-4faf-a425-4ce3e00cec84" volumeName="kubernetes.io/configmap/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-cabundle" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788179 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbf09128-a8c1-4d32-958b-cbd01b892097" volumeName="kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788202 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4588e6fe-4b1e-4e62-8528-7411e598b444" volumeName="kubernetes.io/configmap/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-config" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788211 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" volumeName="kubernetes.io/projected/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-kube-api-access-7vgmg" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788219 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3" volumeName="kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-config" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788226 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56fd7c23-5b71-424f-87e6-91f34918efe8" volumeName="kubernetes.io/projected/56fd7c23-5b71-424f-87e6-91f34918efe8-kube-api-access-fjxrh" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788233 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806fd3b6-00de-4f42-8c35-e638d3835ec7" volumeName="kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-bound-sa-token" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788240 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a310cb37-d20b-4b25-9796-c5f56c44adac" volumeName="kubernetes.io/projected/a310cb37-d20b-4b25-9796-c5f56c44adac-kube-api-access-jblp7" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788248 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a310cb37-d20b-4b25-9796-c5f56c44adac" volumeName="kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-stats-auth" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788256 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3" volumeName="kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-env-overrides" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788263 5055 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3db07759-c867-4de9-84c7-24d903510dca" volumeName="kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-tmp" seLinuxMountContext="" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788270 5055 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:17:12.788780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.788275 5055 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:17:12.791585 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.791565 5055 factory.go:153] Registering CRI-O factory Oct 30 01:17:12.791712 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.791590 5055 factory.go:223] Registration of the crio container factory successfully Oct 30 01:17:12.791712 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.791652 5055 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:17:12.791712 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.791676 5055 factory.go:103] Registering Raw factory Oct 30 01:17:12.791712 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.791691 5055 manager.go:1199] Started watching for new ooms in manager Oct 30 01:17:12.792226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.792197 5055 manager.go:319] Starting recovery of all containers Oct 30 01:17:12.808242 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.808214 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/1.log" Oct 30 01:17:12.809192 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.809169 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/1.log" Oct 30 01:17:12.810370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.810344 5055 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:17:12.811703 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.811687 5055 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:17:12.811703 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.811705 5055 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:17:12.811846 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.811723 5055 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:17:12.811846 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:12.811765 5055 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 01:17:12.830868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.830804 5055 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 01:17:12.831256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.831224 5055 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:12.842071 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.842050 5055 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:12.843888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.843866 5055 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:12.844151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.844137 5055 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:12.852610 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.852563 5055 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="bb140a901cfd5d99ac10a3734ac47d8976627c8768d3a98d3e7cb6317ce5097c" exitCode=0 Oct 30 01:17:12.852610 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.852596 5055 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="6bdd3ffb117ec1d05af86da3ccae99e968ac457bfe61bf2532baba3ba3f80279" exitCode=0 Oct 30 01:17:12.852610 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.852605 5055 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="2aaccf018137369123e95024f6f7b108e421428647133dc1a47fa4a2040b5e2c" exitCode=0 Oct 30 01:17:12.852794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.852614 5055 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="19c076481d99dc37442b14f1ea8af7c393a52f2fd0d5540785459617513c58cd" exitCode=0 Oct 30 01:17:12.852794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.852623 5055 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="b74fc4b33d2d5ad89222c0f3242597fd72c63adfe79bcede2ee4c63a86b73204" exitCode=0 Oct 30 01:17:12.852794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.852630 5055 generic.go:356] "Generic (PLEG): container finished" podID="335e4ad1-7154-4e2c-a170-d52aee2df045" containerID="12e01038f8fadcbb2def5a45bc19095d0ae4692f670b50ca0b7234c7552bafdb" exitCode=0 Oct 30 01:17:12.854516 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.854497 5055 generic.go:356] "Generic (PLEG): container finished" podID="26f52b57e04740fa6edc4f85ae11f68f" containerID="a38bfbd5a97bb05a3b901a2adb00f175951d9bf279d4f773ef345d9528825b38" exitCode=0 Oct 30 01:17:12.862409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.862386 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/1.log" Oct 30 01:17:12.862512 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.862413 5055 generic.go:356] "Generic (PLEG): container finished" podID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" containerID="d3e27a8324376e2d5c186fcfa676d1a7925a6e551c97b7f07801c298375c6438" exitCode=255 Oct 30 01:17:12.892695 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.892672 5055 manager.go:324] Recovery completed Oct 30 01:17:12.911834 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:12.911809 5055 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 01:17:12.916855 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.916836 5055 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:17:12.916855 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.916852 5055 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:17:12.916855 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.916868 5055 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:17:12.917012 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.916978 5055 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 01:17:12.917012 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.916989 5055 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 01:17:12.917117 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.917093 5055 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 01:17:12.917117 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.917099 5055 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 01:17:12.917117 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.917103 5055 policy_none.go:50] "Start" Oct 30 01:17:12.917117 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.917111 5055 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:17:12.917244 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.917120 5055 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:17:12.917276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.917252 5055 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 01:17:12.917276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.917260 5055 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 01:17:12.917276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.917265 5055 policy_none.go:44] "Start" Oct 30 01:17:12.921582 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.921555 5055 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:17:12.924098 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.924076 5055 manager.go:341] "Starting Device Plugin manager" Oct 30 01:17:12.924757 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:12.924100 5055 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:17:12.924827 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.924762 5055 server.go:87] "Starting device plugin registration server" Oct 30 01:17:12.925090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.925077 5055 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:17:12.925136 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.925094 5055 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:17:12.925197 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.925177 5055 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:17:12.925354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.925295 5055 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:17:12.925437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.925355 5055 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:17:12.925437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.925362 5055 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:17:12.925533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.925443 5055 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:17:12.925343775Z","UUID":"713413c9-6d0d-4c30-a6c8-9ddc44c52d41","Handler":null,"Name":"","Endpoint":""} Oct 30 01:17:12.927021 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.927002 5055 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:17:12.927120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:12.927039 5055 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:17:12.927180 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:12.927128 5055 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:17:13.026028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.025994 5055 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:17:13.027312 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.027292 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:17:13.027410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.027346 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:17:13.027410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.027358 5055 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-134-254.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:17:13.027410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.027396 5055 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.064101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.064075 5055 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.064202 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.064160 5055 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.112610 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.112530 5055 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal"] Oct 30 01:17:13.113220 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.113184 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" event={"ID":"7dddf377448d96e272c684d4e526ba4c","Type":"ContainerStarted","Data":"56ae7997392d347539587e80d4f4dd8d876f3782548c1b534cb1d7d53c68d06a"} Oct 30 01:17:13.113220 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.113209 5055 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.113411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.113228 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" event={"ID":"7dddf377448d96e272c684d4e526ba4c","Type":"ContainerStarted","Data":"cdd603789b0b0581f00e91ebf74be898007d44ba6a48597712bde59946488b93"} Oct 30 01:17:13.113411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.113285 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" event={"ID":"26f52b57e04740fa6edc4f85ae11f68f","Type":"ContainerStarted","Data":"b0e3f16e514da64dbaf702b37db6257fa91b16e75dc27bf20f003ca2bc805c8c"} Oct 30 01:17:13.113411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.113304 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" event={"ID":"26f52b57e04740fa6edc4f85ae11f68f","Type":"ContainerDied","Data":"a38bfbd5a97bb05a3b901a2adb00f175951d9bf279d4f773ef345d9528825b38"} Oct 30 01:17:13.113411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.113349 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" event={"ID":"26f52b57e04740fa6edc4f85ae11f68f","Type":"ContainerStarted","Data":"015b60aeb8a694313ddfb327651057a6e9449ec69363a98c0b78412de46af90c"} Oct 30 01:17:13.113532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.113435 5055 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.137791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.137773 5055 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:13.137866 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:13.137808 5055 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-254.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.140554 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.140541 5055 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:13.140599 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:13.140574 5055 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.232822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.232799 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.232921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.232827 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.232921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.232843 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dddf377448d96e272c684d4e526ba4c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-254.ec2.internal\" (UID: \"7dddf377448d96e272c684d4e526ba4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.333440 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.333409 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.333440 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.333433 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.333606 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.333468 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.333606 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.333502 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dddf377448d96e272c684d4e526ba4c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-254.ec2.internal\" (UID: \"7dddf377448d96e272c684d4e526ba4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.333606 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.333551 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7dddf377448d96e272c684d4e526ba4c-config\") pod \"kube-apiserver-proxy-ip-10-0-134-254.ec2.internal\" (UID: \"7dddf377448d96e272c684d4e526ba4c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.333606 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.333561 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26f52b57e04740fa6edc4f85ae11f68f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" (UID: \"26f52b57e04740fa6edc4f85ae11f68f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.768933 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.768903 5055 apiserver.go:52] "Watching apiserver" Oct 30 01:17:13.810947 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.810916 5055 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:17:13.811780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.811749 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v","openshift-ingress/router-default-67bf8c5c5-mtkk5","openshift-image-registry/image-registry-54db9d5677-ptwtb","openshift-image-registry/node-ca-kckpv","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb","openshift-multus/multus-additional-cni-plugins-v4slr","kube-system/kas-connection-checker-685895c7c9-c9s9t","openshift-network-console/networking-console-plugin-86c478bb88-kzdph","openshift-image-registry/image-registry-655b484568-p27mb","kube-system/konnectivity-agent-mvmkl","openshift-ingress-canary/ingress-canary-lh2vb","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg","openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf","openshift-multus/network-metrics-daemon-p9qrg","kube-system/global-pull-secret-syncer-rz57q","openshift-insights/periodic-gathering-gmfkz-7g8gt","openshift-insights/insights-runtime-extractor-jz2gt","openshift-network-diagnostics/network-check-target-4cnv8","openshift-monitoring/cluster-monitoring-operator-884576c57-pb877","openshift-dns/node-resolver-9gckm","openshift-multus/multus-nvmrz","kube-system/kas-connection-checker-685895c7c9-xsnwp","openshift-dns/dns-default-qdpcp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal","openshift-console-operator/console-operator-7cd56b8dbc-tw27b","openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26","openshift-cluster-node-tuning-operator/tuned-f4chz","openshift-network-operator/iptables-alerter-blx29","openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv","kube-system/kas-connection-checker-685895c7c9-4q66s","kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal","openshift-insights/insights-operator-7c4986d449-64j9l","openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm","openshift-ovn-kubernetes/ovnkube-node-vx2b8","openshift-service-ca/service-ca-5fffc97bdf-2c2hh"] Oct 30 01:17:13.813512 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.813495 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:13.813988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.813965 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:13.820539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.820510 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:13.820613 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.820514 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.821413 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.821394 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:13.823409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.823393 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:17:13.826921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.826904 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.827399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.827386 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:13.827636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.827621 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.827713 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.827648 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.828694 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.828672 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qgx7d\"" Oct 30 01:17:13.828977 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.828960 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 01:17:13.829067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829022 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 01:17:13.829132 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829089 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.829132 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829108 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 01:17:13.829132 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829118 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 01:17:13.829262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829136 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.829262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829138 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 01:17:13.829262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829152 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 01:17:13.829262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829153 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.829262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829171 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zbwdc\"" Oct 30 01:17:13.829262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829196 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.829262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829223 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 01:17:13.829609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.829388 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:13.836064 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.836046 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.836589 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.836575 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.837776 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.837753 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 01:17:13.837873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.837835 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xd5g5\"" Oct 30 01:17:13.837873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.837859 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 01:17:13.837964 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.837877 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 01:17:13.837964 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.837895 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 01:17:13.838033 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.837984 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cwfw8\"" Oct 30 01:17:13.838033 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.837987 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-f5n9z\"" Oct 30 01:17:13.844025 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.844011 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-58v5x\"" Oct 30 01:17:13.844871 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.844849 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:17:13.845495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.845467 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 01:17:13.845666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.845652 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:17:13.845736 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.845691 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:17:13.845789 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.845778 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xp5gx\"" Oct 30 01:17:13.845879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.845859 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:17:13.845935 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.845877 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7dsj7\"" Oct 30 01:17:13.845935 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.845868 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tgxzm\"" Oct 30 01:17:13.845935 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.845933 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 01:17:13.846065 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.845987 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lvp69\"" Oct 30 01:17:13.846065 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.846004 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:17:13.846755 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.846548 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p4hjp\"" Oct 30 01:17:13.847289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.847267 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 01:17:13.848206 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.848184 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wjscp\"" Oct 30 01:17:13.848572 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.848551 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2b59\"" Oct 30 01:17:13.848685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.848667 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xmk59\"" Oct 30 01:17:13.848788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.848773 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:17:13.848835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.848823 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-9gpf8\"" Oct 30 01:17:13.848905 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.848888 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.849663 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.849642 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 01:17:13.849830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.849807 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 01:17:13.849944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.849927 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:17:13.850080 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850067 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.850128 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850078 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.850171 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850137 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pcscp\"" Oct 30 01:17:13.850216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850167 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.850302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850285 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-g5b5q\"" Oct 30 01:17:13.850434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850414 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-srg4h\"" Oct 30 01:17:13.850500 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850481 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n79rv\"" Oct 30 01:17:13.850595 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850579 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-58lhf\"" Oct 30 01:17:13.850726 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850712 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4mfm\"" Oct 30 01:17:13.850772 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850753 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms58v\"" Oct 30 01:17:13.850989 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.850971 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:17:13.851177 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851161 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:17:13.851306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851291 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 01:17:13.851394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851305 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:17:13.851394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851361 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.851394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851309 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8bsj\"" Oct 30 01:17:13.851525 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851405 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mf6zs\"" Oct 30 01:17:13.851525 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851448 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:17:13.851525 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851357 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 01:17:13.851525 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851508 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.851750 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851566 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.851750 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851369 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:17:13.851750 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851647 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.851750 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851680 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:17:13.851750 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851310 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:17:13.851970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851782 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.851970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851788 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.851970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851823 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.851970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851824 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.851970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851858 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.851970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851894 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.851970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851895 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.851970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851944 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.851970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851947 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:17:13.851970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851954 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.852343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.851994 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:17:13.852343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852046 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.852343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852052 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.852343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852137 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.852343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852146 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 01:17:13.852343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852166 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.852553 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852413 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.852553 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852454 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.852770 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852755 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zs626\"" Oct 30 01:17:13.852817 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852810 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hb7tb\"" Oct 30 01:17:13.852857 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852847 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fsgft\"" Oct 30 01:17:13.852908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852896 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 01:17:13.852982 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.852971 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.853096 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.853084 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.853139 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.853120 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.853178 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.853147 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.856530 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.856514 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.856630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.856529 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.856630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.856579 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.856630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.856591 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:17:13.856724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.856619 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:17:13.858574 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.858558 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:17:13.858738 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.858725 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 01:17:13.858843 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.858829 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 01:17:13.861870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.861841 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 01:17:13.862756 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.862734 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 01:17:13.862756 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.862761 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:17:13.862913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.862856 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 01:17:13.862913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.862864 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 01:17:13.862913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.862857 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 01:17:13.862913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.862906 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 01:17:13.863263 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.863246 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5t49s\"" Oct 30 01:17:13.875063 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.875047 5055 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.875159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.875146 5055 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.883779 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.883762 5055 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:17:13.895533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.895518 5055 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:13.895622 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:13.895555 5055 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-134-254.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.896028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.896017 5055 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:13.896065 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:13.896045 5055 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-254.ec2.internal" Oct 30 01:17:13.935480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935444 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:13.935480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935477 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrqvr\" (UniqueName: \"kubernetes.io/projected/b47f8e4a-5060-47bf-8bec-722ccff0fa51-kube-api-access-hrqvr\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:17:13.935644 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935499 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-slash\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.935644 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935515 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-env-overrides\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.935644 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935536 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-certificates\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.935644 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935560 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysconfig\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.935644 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935599 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-sys\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.935881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935685 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-netns\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.935881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935717 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.935881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935781 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-env-overrides\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.935881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935852 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/806fd3b6-00de-4f42-8c35-e638d3835ec7-ca-trust-extracted\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.935881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935860 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8bc436c0-2b1b-477e-81aa-af1a1eff9779-original-pull-secret\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:13.936070 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.935883 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/083c7373-177e-46cb-99c1-f9509121668a-serving-cert\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.936070 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936031 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/806fd3b6-00de-4f42-8c35-e638d3835ec7-ca-trust-extracted\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.936140 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936123 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:13.936205 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936184 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-system-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.936239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936223 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-etc-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.936288 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936272 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovn-node-metrics-cert\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.936341 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936305 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-cnibin\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.936392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936357 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8q47\" (UniqueName: \"kubernetes.io/projected/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-kube-api-access-f8q47\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:13.936392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936360 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.936392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936382 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-host\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.936485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936430 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/083c7373-177e-46cb-99c1-f9509121668a-serving-cert\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.936485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936435 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-certificates\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.936485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936462 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-trusted-ca\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.936579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936503 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-snapshots\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.936579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936528 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-trusted-ca\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.936579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936552 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-config-volume\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:13.936579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936567 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-certificates\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.936710 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936567 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovn-node-metrics-cert\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.936710 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936623 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/184de02f-0273-408d-9ee5-265a0bb278a4-data-volume\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:13.936710 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936650 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/712bffb7-97f2-422a-ba7b-cce9eafccba4-serving-cert\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:13.936802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936711 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-multus\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.936802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936736 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-var-lib-kubelet\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.936802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936773 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-registration-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:13.936802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936786 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-config-volume\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:13.936971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936804 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-tmp\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.936971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936909 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/184de02f-0273-408d-9ee5-265a0bb278a4-data-volume\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:13.936971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936910 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-tmp\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.937087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.936973 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf4dt\" (UniqueName: \"kubernetes.io/projected/184de02f-0273-408d-9ee5-265a0bb278a4-kube-api-access-mf4dt\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:13.937087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937005 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cni-binary-copy\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.937087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937037 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-socket-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:13.937180 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937066 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.937180 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937114 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-installation-pull-secrets\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.937180 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937144 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.937286 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937263 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cnibin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.937320 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937288 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-systemd-units\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.937320 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937310 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-conf\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.937455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937358 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7cmp\" (UniqueName: \"kubernetes.io/projected/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-kube-api-access-h7cmp\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:13.937455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937363 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-trusted-ca\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.937455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937381 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/712bffb7-97f2-422a-ba7b-cce9eafccba4-serving-cert\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:13.937455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937388 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jwhg\" (UniqueName: \"kubernetes.io/projected/bb940654-4320-4340-a2d5-9450bc68f40f-kube-api-access-6jwhg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:13.937455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937437 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9b752e3-e38a-4d0f-babd-7922d6a2e872-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:13.937696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937456 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/083c7373-177e-46cb-99c1-f9509121668a-snapshots\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.937696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937467 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwz2\" (UniqueName: \"kubernetes.io/projected/c9b752e3-e38a-4d0f-babd-7922d6a2e872-kube-api-access-gnwz2\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:13.937696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937492 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-trusted-ca\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.937696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937503 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzmb8\" (UniqueName: \"kubernetes.io/projected/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-kube-api-access-bzmb8\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:13.937696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937563 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-kubelet\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.937696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937589 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-daemon-config\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.937696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937613 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-netd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.937696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937638 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-script-lib\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.938066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937707 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-trusted-ca-bundle\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:13.938066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937789 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7082f68a-2079-426e-997f-1d3ba3997ba6-ca-trust-extracted\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.938066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937799 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-installation-pull-secrets\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.938066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937828 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:13.938066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937859 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b47f8e4a-5060-47bf-8bec-722ccff0fa51-hosts-file\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:17:13.938066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937975 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:13.938066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938028 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:13.938066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938058 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgmg\" (UniqueName: \"kubernetes.io/projected/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-kube-api-access-7vgmg\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:13.938463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938108 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-service-ca-bundle\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.938463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938119 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-script-lib\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.938463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938139 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7082f68a-2079-426e-997f-1d3ba3997ba6-ca-trust-extracted\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.938463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938237 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cni-binary-copy\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.938463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.937981 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-trusted-ca-bundle\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:13.938463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938301 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-daemon-config\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.938463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938351 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-certificates\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.938463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938459 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/c9b752e3-e38a-4d0f-babd-7922d6a2e872-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:13.938803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938494 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-bin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.938803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938527 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-modprobe-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.938803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938565 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.938803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938600 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-config\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:13.938803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938639 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/628af330-6652-4df0-8275-9f691e2c8eb3-konnectivity-ca\") pod \"konnectivity-agent-mvmkl\" (UID: \"628af330-6652-4df0-8275-9f691e2c8eb3\") " pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:17:13.938803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938673 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25s9d\" (UniqueName: \"kubernetes.io/projected/f207f3a7-cf1c-4acc-8847-267e7c89e31f-kube-api-access-25s9d\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:17:13.938803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938703 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k4bh\" (UniqueName: \"kubernetes.io/projected/b4e79600-74c8-4e22-8fce-fbd02aca1e38-kube-api-access-6k4bh\") pod \"kas-connection-checker-685895c7c9-xsnwp\" (UID: \"b4e79600-74c8-4e22-8fce-fbd02aca1e38\") " pod="kube-system/kas-connection-checker-685895c7c9-xsnwp" Oct 30 01:17:13.938803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938705 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-service-ca-bundle\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.938803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938764 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-run\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.939145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938823 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jb47\" (UniqueName: \"kubernetes.io/projected/4588e6fe-4b1e-4e62-8528-7411e598b444-kube-api-access-9jb47\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:13.939145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938884 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-var-lib-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.939145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938912 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-bin\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.939145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.938985 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.939145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939024 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/628af330-6652-4df0-8275-9f691e2c8eb3-agent-certs\") pod \"konnectivity-agent-mvmkl\" (UID: \"628af330-6652-4df0-8275-9f691e2c8eb3\") " pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:17:13.939145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939066 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-config\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:13.939145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939072 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f207f3a7-cf1c-4acc-8847-267e7c89e31f-iptables-alerter-script\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:17:13.939145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939096 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/628af330-6652-4df0-8275-9f691e2c8eb3-konnectivity-ca\") pod \"konnectivity-agent-mvmkl\" (UID: \"628af330-6652-4df0-8275-9f691e2c8eb3\") " pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:17:13.939145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939115 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46q9r\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-kube-api-access-46q9r\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.939145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939144 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6khjk\" (UniqueName: \"kubernetes.io/projected/712bffb7-97f2-422a-ba7b-cce9eafccba4-kube-api-access-6khjk\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939170 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-service-ca-bundle\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939192 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-default-certificate\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939214 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqtw\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-kube-api-access-2tqtw\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939258 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-socket-dir-parent\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939285 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx866\" (UniqueName: \"kubernetes.io/projected/495d54bf-6c21-46ab-badd-8ece131d6618-kube-api-access-sx866\") pod \"network-check-source-6f78d59f99-rgvjv\" (UID: \"495d54bf-6c21-46ab-badd-8ece131d6618\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939311 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939345 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/628af330-6652-4df0-8275-9f691e2c8eb3-agent-certs\") pod \"konnectivity-agent-mvmkl\" (UID: \"628af330-6652-4df0-8275-9f691e2c8eb3\") " pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939358 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-conf-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939359 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f207f3a7-cf1c-4acc-8847-267e7c89e31f-iptables-alerter-script\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939404 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-etc-kubernetes\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939528 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939574 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-default-certificate\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:13.939601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939583 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939672 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-nginx-conf\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939700 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-dbus\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939723 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-system-cni-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939742 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939766 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9801105-8f8a-45d0-b2c1-23468a212347-serviceca\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939808 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-kubelet\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939821 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/083c7373-177e-46cb-99c1-f9509121668a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939865 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-systemd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939862 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-service-ca-bundle\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939901 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb940654-4320-4340-a2d5-9450bc68f40f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939902 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.939935 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcv6t\" (UniqueName: \"kubernetes.io/projected/98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db-kube-api-access-vcv6t\") pod \"volume-data-source-validator-58c9f5f46d-z5d5r\" (UID: \"98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940048 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940127 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-k8s-cni-cncf-io\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940185 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-image-registry-private-configuration\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.940193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940186 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb940654-4320-4340-a2d5-9450bc68f40f-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940218 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940238 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47f8e4a-5060-47bf-8bec-722ccff0fa51-tmp-dir\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940266 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-ovn\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940282 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-config\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940297 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940312 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940349 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b47f8e4a-5060-47bf-8bec-722ccff0fa51-tmp-dir\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940353 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqbqr\" (UniqueName: \"kubernetes.io/projected/335e4ad1-7154-4e2c-a170-d52aee2df045-kube-api-access-dqbqr\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940377 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f207f3a7-cf1c-4acc-8847-267e7c89e31f-host-slash\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940417 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-image-registry-private-configuration\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940466 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c9801105-8f8a-45d0-b2c1-23468a212347-serviceca\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940517 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2cf9\" (UniqueName: \"kubernetes.io/projected/8855c2e7-9182-4ae6-b96a-50baa69d9a67-kube-api-access-z2cf9\") pod \"ingress-canary-lh2vb\" (UID: \"8855c2e7-9182-4ae6-b96a-50baa69d9a67\") " pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940534 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-ovnkube-config\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940545 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p5mm\" (UniqueName: \"kubernetes.io/projected/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-kube-api-access-9p5mm\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940567 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/335e4ad1-7154-4e2c-a170-d52aee2df045-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940577 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940614 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-serving-cert\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940644 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940698 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jblp7\" (UniqueName: \"kubernetes.io/projected/a310cb37-d20b-4b25-9796-c5f56c44adac-kube-api-access-jblp7\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940741 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-tmp-dir\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940769 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940773 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-serving-cert\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940797 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzwm\" (UniqueName: \"kubernetes.io/projected/083c7373-177e-46cb-99c1-f9509121668a-kube-api-access-8nzwm\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940829 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9801105-8f8a-45d0-b2c1-23468a212347-host\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940832 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-tmp-dir\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940858 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/184de02f-0273-408d-9ee5-265a0bb278a4-crio-socket\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940902 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-log-socket\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.940914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940921 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940966 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-hostroot\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.940990 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-lib-modules\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941015 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-key\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941051 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2529z\" (UniqueName: \"kubernetes.io/projected/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-kube-api-access-2529z\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941184 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-image-registry-private-configuration\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941229 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-key\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941270 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-cabundle\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941305 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-node-log\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941371 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-image-registry-private-configuration\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941369 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-trusted-ca\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941406 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-bound-sa-token\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941434 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941489 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/712bffb7-97f2-422a-ba7b-cce9eafccba4-config\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941502 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-signing-cabundle\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941579 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-netns\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.941944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941919 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-trusted-ca\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:13.942620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.941957 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-multus-certs\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.942620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.942016 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-kubernetes\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.942620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.942059 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:13.942620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.942207 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-etc-tuned\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.942620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.942281 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-tmp\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.942620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.942297 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:13.942620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.942426 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-tmp\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.942620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.942459 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/712bffb7-97f2-422a-ba7b-cce9eafccba4-config\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:13.942620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.942480 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3db07759-c867-4de9-84c7-24d903510dca-etc-tuned\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.942620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.942550 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-archives-path\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.942316 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-archives-path\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943105 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb940654-4320-4340-a2d5-9450bc68f40f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943135 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-bound-sa-token\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943168 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943203 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-sys-fs\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943238 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pzfs\" (UniqueName: \"kubernetes.io/projected/1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5-kube-api-access-8pzfs\") pod \"kas-connection-checker-685895c7c9-4q66s\" (UID: \"1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5\") " pod="kube-system/kas-connection-checker-685895c7c9-4q66s" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943272 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwp4f\" (UniqueName: \"kubernetes.io/projected/4a8853b4-1b14-4b16-81d8-007e7827d8dd-kube-api-access-lwp4f\") pod \"kas-connection-checker-685895c7c9-c9s9t\" (UID: \"4a8853b4-1b14-4b16-81d8-007e7827d8dd\") " pod="kube-system/kas-connection-checker-685895c7c9-c9s9t" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943301 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-stats-auth\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943360 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4d5n\" (UniqueName: \"kubernetes.io/projected/cc0b4765-071a-4af8-88e6-700abd1d8f83-kube-api-access-k4d5n\") pod \"migrator-66df5d496b-jgj26\" (UID: \"cc0b4765-071a-4af8-88e6-700abd1d8f83\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943392 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxrh\" (UniqueName: \"kubernetes.io/projected/56fd7c23-5b71-424f-87e6-91f34918efe8-kube-api-access-fjxrh\") pod \"migrator-66df5d496b-jj7dm\" (UID: \"56fd7c23-5b71-424f-87e6-91f34918efe8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943423 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943473 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-device-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.943779 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-stats-auth\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.944081 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bb940654-4320-4340-a2d5-9450bc68f40f-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.944201 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwd46\" (UniqueName: \"kubernetes.io/projected/d8c54d89-eea4-45ee-80e5-fd0aba53e274-kube-api-access-nwd46\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.944243 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trgqt\" (UniqueName: \"kubernetes.io/projected/c9801105-8f8a-45d0-b2c1-23468a212347-kube-api-access-trgqt\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.944274 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8855c2e7-9182-4ae6-b96a-50baa69d9a67-cert\") pod \"ingress-canary-lh2vb\" (UID: \"8855c2e7-9182-4ae6-b96a-50baa69d9a67\") " pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.944402 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.944433 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-systemd\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.944469 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29vzs\" (UniqueName: \"kubernetes.io/projected/3db07759-c867-4de9-84c7-24d903510dca-kube-api-access-29vzs\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.944508 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzcm2\" (UniqueName: \"kubernetes.io/projected/97b353f9-eeeb-4f6a-bdb7-204450f76167-kube-api-access-bzcm2\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.944587 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-installation-pull-secrets\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.944622 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-kubelet-config\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.945126 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8855c2e7-9182-4ae6-b96a-50baa69d9a67-cert\") pod \"ingress-canary-lh2vb\" (UID: \"8855c2e7-9182-4ae6-b96a-50baa69d9a67\") " pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:13.946559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.946017 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-installation-pull-secrets\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.955437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.955409 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrqvr\" (UniqueName: \"kubernetes.io/projected/b47f8e4a-5060-47bf-8bec-722ccff0fa51-kube-api-access-hrqvr\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:17:13.955511 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.955411 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8q47\" (UniqueName: \"kubernetes.io/projected/b6cb0d17-b098-408e-92b6-b5e81a8bb9cf-kube-api-access-f8q47\") pod \"console-operator-7cd56b8dbc-tw27b\" (UID: \"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:13.961911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.961554 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf4dt\" (UniqueName: \"kubernetes.io/projected/184de02f-0273-408d-9ee5-265a0bb278a4-kube-api-access-mf4dt\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:13.961911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.961693 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jwhg\" (UniqueName: \"kubernetes.io/projected/bb940654-4320-4340-a2d5-9450bc68f40f-kube-api-access-6jwhg\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v\" (UID: \"bb940654-4320-4340-a2d5-9450bc68f40f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" Oct 30 01:17:13.961911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.961760 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzmb8\" (UniqueName: \"kubernetes.io/projected/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-kube-api-access-bzmb8\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:13.961911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.961777 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7cmp\" (UniqueName: \"kubernetes.io/projected/9f4577e2-2c6f-4faf-a425-4ce3e00cec84-kube-api-access-h7cmp\") pod \"service-ca-5fffc97bdf-2c2hh\" (UID: \"9f4577e2-2c6f-4faf-a425-4ce3e00cec84\") " pod="openshift-service-ca/service-ca-5fffc97bdf-2c2hh" Oct 30 01:17:13.961911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.961843 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25s9d\" (UniqueName: \"kubernetes.io/projected/f207f3a7-cf1c-4acc-8847-267e7c89e31f-kube-api-access-25s9d\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:17:13.962218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.962023 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgmg\" (UniqueName: \"kubernetes.io/projected/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-kube-api-access-7vgmg\") pod \"periodic-gathering-gmfkz-7g8gt\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:17:13.962218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.962036 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6khjk\" (UniqueName: \"kubernetes.io/projected/712bffb7-97f2-422a-ba7b-cce9eafccba4-kube-api-access-6khjk\") pod \"service-ca-operator-544d749c8c-tm7mf\" (UID: \"712bffb7-97f2-422a-ba7b-cce9eafccba4\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" Oct 30 01:17:13.962402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.962312 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p5mm\" (UniqueName: \"kubernetes.io/projected/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-kube-api-access-9p5mm\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:13.962402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.962395 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46q9r\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-kube-api-access-46q9r\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.962533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.962412 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqtw\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-kube-api-access-2tqtw\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.962533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.962428 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwz2\" (UniqueName: \"kubernetes.io/projected/c9b752e3-e38a-4d0f-babd-7922d6a2e872-kube-api-access-gnwz2\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:13.962816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.962800 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx866\" (UniqueName: \"kubernetes.io/projected/495d54bf-6c21-46ab-badd-8ece131d6618-kube-api-access-sx866\") pod \"network-check-source-6f78d59f99-rgvjv\" (UID: \"495d54bf-6c21-46ab-badd-8ece131d6618\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-rgvjv" Oct 30 01:17:13.962881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.962863 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqbqr\" (UniqueName: \"kubernetes.io/projected/335e4ad1-7154-4e2c-a170-d52aee2df045-kube-api-access-dqbqr\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:13.962998 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.962983 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcv6t\" (UniqueName: \"kubernetes.io/projected/98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db-kube-api-access-vcv6t\") pod \"volume-data-source-validator-58c9f5f46d-z5d5r\" (UID: \"98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-z5d5r" Oct 30 01:17:13.964501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.964486 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2cf9\" (UniqueName: \"kubernetes.io/projected/8855c2e7-9182-4ae6-b96a-50baa69d9a67-kube-api-access-z2cf9\") pod \"ingress-canary-lh2vb\" (UID: \"8855c2e7-9182-4ae6-b96a-50baa69d9a67\") " pod="openshift-ingress-canary/ingress-canary-lh2vb" Oct 30 01:17:13.964608 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.964592 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2529z\" (UniqueName: \"kubernetes.io/projected/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-kube-api-access-2529z\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:13.968317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968082 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-bound-sa-token\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:13.968317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968118 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4d5n\" (UniqueName: \"kubernetes.io/projected/cc0b4765-071a-4af8-88e6-700abd1d8f83-kube-api-access-k4d5n\") pod \"migrator-66df5d496b-jgj26\" (UID: \"cc0b4765-071a-4af8-88e6-700abd1d8f83\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jgj26" Oct 30 01:17:13.968317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968174 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29vzs\" (UniqueName: \"kubernetes.io/projected/3db07759-c867-4de9-84c7-24d903510dca-kube-api-access-29vzs\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:13.968317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968265 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jblp7\" (UniqueName: \"kubernetes.io/projected/a310cb37-d20b-4b25-9796-c5f56c44adac-kube-api-access-jblp7\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:13.968317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968307 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzwm\" (UniqueName: \"kubernetes.io/projected/083c7373-177e-46cb-99c1-f9509121668a-kube-api-access-8nzwm\") pod \"insights-operator-7c4986d449-64j9l\" (UID: \"083c7373-177e-46cb-99c1-f9509121668a\") " pod="openshift-insights/insights-operator-7c4986d449-64j9l" Oct 30 01:17:13.968657 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968597 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k4bh\" (UniqueName: \"kubernetes.io/projected/b4e79600-74c8-4e22-8fce-fbd02aca1e38-kube-api-access-6k4bh\") pod \"kas-connection-checker-685895c7c9-xsnwp\" (UID: \"b4e79600-74c8-4e22-8fce-fbd02aca1e38\") " pod="kube-system/kas-connection-checker-685895c7c9-xsnwp" Oct 30 01:17:13.968657 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968618 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq8g2\" (UniqueName: \"kubernetes.io/projected/dbf09128-a8c1-4d32-958b-cbd01b892097-kube-api-access-qq8g2\") pod \"network-check-target-4cnv8\" (UID: \"dbf09128-a8c1-4d32-958b-cbd01b892097\") " pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:17:13.968730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968674 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-bound-sa-token\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:13.968786 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968765 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pzfs\" (UniqueName: \"kubernetes.io/projected/1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5-kube-api-access-8pzfs\") pod \"kas-connection-checker-685895c7c9-4q66s\" (UID: \"1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5\") " pod="kube-system/kas-connection-checker-685895c7c9-4q66s" Oct 30 01:17:13.968958 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968939 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jb47\" (UniqueName: \"kubernetes.io/projected/4588e6fe-4b1e-4e62-8528-7411e598b444-kube-api-access-9jb47\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:13.969047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.968974 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwp4f\" (UniqueName: \"kubernetes.io/projected/4a8853b4-1b14-4b16-81d8-007e7827d8dd-kube-api-access-lwp4f\") pod \"kas-connection-checker-685895c7c9-c9s9t\" (UID: \"4a8853b4-1b14-4b16-81d8-007e7827d8dd\") " pod="kube-system/kas-connection-checker-685895c7c9-c9s9t" Oct 30 01:17:13.969259 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.969242 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwd46\" (UniqueName: \"kubernetes.io/projected/d8c54d89-eea4-45ee-80e5-fd0aba53e274-kube-api-access-nwd46\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:17:13.969348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.969296 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzcm2\" (UniqueName: \"kubernetes.io/projected/97b353f9-eeeb-4f6a-bdb7-204450f76167-kube-api-access-bzcm2\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:13.969348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.969315 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgqt\" (UniqueName: \"kubernetes.io/projected/c9801105-8f8a-45d0-b2c1-23468a212347-kube-api-access-trgqt\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:17:13.969418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:13.969372 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxrh\" (UniqueName: \"kubernetes.io/projected/56fd7c23-5b71-424f-87e6-91f34918efe8-kube-api-access-fjxrh\") pod \"migrator-66df5d496b-jj7dm\" (UID: \"56fd7c23-5b71-424f-87e6-91f34918efe8\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-jj7dm" Oct 30 01:17:14.045136 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045068 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:14.045136 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045099 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-k8s-cni-cncf-io\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.045136 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045116 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045143 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-ovn\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045168 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045192 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f207f3a7-cf1c-4acc-8847-267e7c89e31f-host-slash\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045194 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-k8s-cni-cncf-io\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045215 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045223 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-ovn\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045238 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045271 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-etc-selinux\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045277 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045284 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f207f3a7-cf1c-4acc-8847-267e7c89e31f-host-slash\") pod \"iptables-alerter-blx29\" (UID: \"f207f3a7-cf1c-4acc-8847-267e7c89e31f\") " pod="openshift-network-operator/iptables-alerter-blx29" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045303 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9801105-8f8a-45d0-b2c1-23468a212347-host\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:17:14.045350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045348 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/184de02f-0273-408d-9ee5-265a0bb278a4-crio-socket\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045372 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-log-socket\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045394 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-hostroot\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045407 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9801105-8f8a-45d0-b2c1-23468a212347-host\") pod \"node-ca-kckpv\" (UID: \"c9801105-8f8a-45d0-b2c1-23468a212347\") " pod="openshift-image-registry/node-ca-kckpv" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045418 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-lib-modules\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045393 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045433 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/184de02f-0273-408d-9ee5-265a0bb278a4-crio-socket\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045446 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-hostroot\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.045438 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:14.545414504 +0000 UTC m=+1.866371522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : configmap references non-existent config key: service-ca.crt Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045471 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-log-socket\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045512 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-node-log\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045495 5055 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045540 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045564 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-netns\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045571 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-lib-modules\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045588 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-multus-certs\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045613 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-kubernetes\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045620 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-netns\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045591 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-node-log\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045643 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-run-multus-certs\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045646 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045681 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-kubernetes\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045688 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-sys-fs\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045750 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045772 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-device-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045796 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045820 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-systemd\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045848 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-kubelet-config\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045862 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-device-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045873 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-slash\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045898 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysconfig\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045817 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-sys-fs\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.045909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045922 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-sys\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045946 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-netns\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045971 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045995 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-system-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046017 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-etc-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046040 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-cnibin\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046063 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-host\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046095 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-multus\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046120 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-var-lib-kubelet\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046143 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-registration-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046171 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-socket-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046194 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046217 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046239 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cnibin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046257 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-system-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046261 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-systemd-units\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046295 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-conf\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046308 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-kubelet-config\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046351 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-kubelet\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046371 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-slash\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046378 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-netd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.045899 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046409 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046422 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-netns\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046437 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b47f8e4a-5060-47bf-8bec-722ccff0fa51-hosts-file\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046462 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046515 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046542 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-bin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046565 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-modprobe-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046606 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046616 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-socket-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046634 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-run\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046660 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-var-lib-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046691 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-bin\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046707 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysconfig\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046716 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046750 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-socket-dir-parent\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046892 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/184de02f-0273-408d-9ee5-265a0bb278a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.046959 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-etc-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047030 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-cnibin\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047126 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-conf-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047157 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-etc-kubernetes\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047186 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047209 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-var-lib-kubelet\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047261 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-multus\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047313 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047276 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-dbus\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047423 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-system-cni-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047451 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-kubelet\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047476 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-systemd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.047569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047573 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-run-systemd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047627 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-cni-bin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047733 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-modprobe-d\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047798 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-run-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047800 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-host\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047853 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-bin\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047892 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-var-lib-openvswitch\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047947 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047982 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/97b353f9-eeeb-4f6a-bdb7-204450f76167-registration-dir\") pod \"aws-ebs-csi-driver-node-6vxjg\" (UID: \"97b353f9-eeeb-4f6a-bdb7-204450f76167\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6vxjg" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.047999 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-socket-dir-parent\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048041 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-conf-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048082 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-etc-kubernetes\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048120 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048175 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8bc436c0-2b1b-477e-81aa-af1a1eff9779-dbus\") pod \"global-pull-secret-syncer-rz57q\" (UID: \"8bc436c0-2b1b-477e-81aa-af1a1eff9779\") " pod="kube-system/global-pull-secret-syncer-rz57q" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048190 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-run\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048375 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-kubelet\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048441 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-systemd\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048489 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-host-var-lib-kubelet\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048547 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-multus-cni-dir\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048600 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17-cnibin\") pod \"multus-nvmrz\" (UID: \"4beb5e05-bee4-4eb3-b205-b4e6bc5d9b17\") " pod="openshift-multus/multus-nvmrz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048645 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-systemd-units\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048745 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a310cb37-d20b-4b25-9796-c5f56c44adac-metrics-certs\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.048739 5055 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048377 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/335e4ad1-7154-4e2c-a170-d52aee2df045-system-cni-dir\") pod \"multus-additional-cni-plugins-v4slr\" (UID: \"335e4ad1-7154-4e2c-a170-d52aee2df045\") " pod="openshift-multus/multus-additional-cni-plugins-v4slr" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048794 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-etc-sysctl-conf\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048802 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4588e6fe-4b1e-4e62-8528-7411e598b444-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-54ddb\" (UID: \"4588e6fe-4b1e-4e62-8528-7411e598b444\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048812 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3-host-cni-netd\") pod \"ovnkube-node-vx2b8\" (UID: \"0b388420-2ed8-4ee3-9f1f-899e2b8ae0c3\") " pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.048824 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls podName:c9b752e3-e38a-4d0f-babd-7922d6a2e872 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:14.548808279 +0000 UTC m=+1.869765320 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pb877" (UID: "c9b752e3-e38a-4d0f-babd-7922d6a2e872") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048820 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b47f8e4a-5060-47bf-8bec-722ccff0fa51-hosts-file\") pod \"node-resolver-9gckm\" (UID: \"b47f8e4a-5060-47bf-8bec-722ccff0fa51\") " pod="openshift-dns/node-resolver-9gckm" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.048859 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3db07759-c867-4de9-84c7-24d903510dca-sys\") pod \"tuned-f4chz\" (UID: \"3db07759-c867-4de9-84c7-24d903510dca\") " pod="openshift-cluster-node-tuning-operator/tuned-f4chz" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.048897 5055 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.048970 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:14.548952731 +0000 UTC m=+1.869909765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : secret "insights-runtime-extractor-tls" not found Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.049166 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8c54d89-eea4-45ee-80e5-fd0aba53e274-metrics-certs\") pod \"network-metrics-daemon-p9qrg\" (UID: \"d8c54d89-eea4-45ee-80e5-fd0aba53e274\") " pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:17:14.049771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.049237 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") pod \"image-registry-655b484568-p27mb\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:14.050724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.049812 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e24a38d-9f35-40bf-bb90-57fee63d8ac0-metrics-tls\") pod \"dns-default-qdpcp\" (UID: \"1e24a38d-9f35-40bf-bb90-57fee63d8ac0\") " pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:14.050724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.050132 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") pod \"image-registry-54db9d5677-ptwtb\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:14.050724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.050439 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-kzdph\" (UID: \"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:14.114038 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.114006 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" Oct 30 01:17:14.117366 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.117344 5055 scope.go:122] "RemoveContainer" containerID="d3e27a8324376e2d5c186fcfa676d1a7925a6e551c97b7f07801c298375c6438" Oct 30 01:17:14.122093 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.122075 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" Oct 30 01:17:14.122242 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.122226 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:14.124148 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.124134 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p9qrg" Oct 30 01:17:14.127484 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.127467 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:14.128530 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.128513 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:14.440082 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.440060 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb"] Oct 30 01:17:14.475200 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.475178 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-kzdph"] Oct 30 01:17:14.475530 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.475514 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:17:14.476790 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:14.476752 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1832b1_6bc7_4bf2_8dd9_de2bd8d55d41.slice/crio-71fc392ed0373fef61066a251b197c41a5b0b12629832cdaf8b3d590c87d8260 WatchSource:0}: Error finding container 71fc392ed0373fef61066a251b197c41a5b0b12629832cdaf8b3d590c87d8260: Status 404 returned error can't find the container with id 71fc392ed0373fef61066a251b197c41a5b0b12629832cdaf8b3d590c87d8260 Oct 30 01:17:14.489999 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.489975 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kckpv_c9801105-8f8a-45d0-b2c1-23468a212347/node-ca/0.log" Oct 30 01:17:14.514764 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.514741 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qdpcp"] Oct 30 01:17:14.516093 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:14.516072 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e24a38d_9f35_40bf_bb90_57fee63d8ac0.slice/crio-0a42a15d22bbadaf051e3e69842abb8e3262565d403da64c24b059936a7e874f WatchSource:0}: Error finding container 0a42a15d22bbadaf051e3e69842abb8e3262565d403da64c24b059936a7e874f: Status 404 returned error can't find the container with id 0a42a15d22bbadaf051e3e69842abb8e3262565d403da64c24b059936a7e874f Oct 30 01:17:14.546091 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.544908 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p9qrg"] Oct 30 01:17:14.551795 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.551748 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:14.551795 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.551781 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:14.551985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.551821 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:14.551985 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.551875 5055 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:14.551985 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.551902 5055 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:17:14.551985 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.551931 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:15.551914275 +0000 UTC m=+2.872871294 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : configmap references non-existent config key: service-ca.crt Oct 30 01:17:14.551985 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.551958 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:15.551940034 +0000 UTC m=+2.872897053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : secret "insights-runtime-extractor-tls" not found Oct 30 01:17:14.551985 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.551977 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls podName:c9b752e3-e38a-4d0f-babd-7922d6a2e872 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:15.551967114 +0000 UTC m=+2.872924138 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pb877" (UID: "c9b752e3-e38a-4d0f-babd-7922d6a2e872") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:14.590894 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.590869 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-655b484568-p27mb"] Oct 30 01:17:14.591705 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:14.591683 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7082f68a_2079_426e_997f_1d3ba3997ba6.slice/crio-a879ec3a72661efac01ed9379c5a826dae53540bb3c8e2ee36adfe094cffa0c0 WatchSource:0}: Error finding container a879ec3a72661efac01ed9379c5a826dae53540bb3c8e2ee36adfe094cffa0c0: Status 404 returned error can't find the container with id a879ec3a72661efac01ed9379c5a826dae53540bb3c8e2ee36adfe094cffa0c0 Oct 30 01:17:14.624072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.624052 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-54db9d5677-ptwtb"] Oct 30 01:17:14.625066 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:14.625045 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806fd3b6_00de_4f42_8c35_e638d3835ec7.slice/crio-25be08911bf0ee43bd80e546c4a59e397e8fe230dfcbc2d7647418f28dec9cff WatchSource:0}: Error finding container 25be08911bf0ee43bd80e546c4a59e397e8fe230dfcbc2d7647418f28dec9cff: Status 404 returned error can't find the container with id 25be08911bf0ee43bd80e546c4a59e397e8fe230dfcbc2d7647418f28dec9cff Oct 30 01:17:14.662187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.662163 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:17:14.881682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.881609 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:17:14.882159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.882136 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/1.log" Oct 30 01:17:14.882280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.882176 5055 generic.go:356] "Generic (PLEG): container finished" podID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" containerID="bb047c1ffd10df71ec667eaacfdc46766e3f8b2a469d70cc27c5d32aec157204" exitCode=255 Oct 30 01:17:14.882280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.882264 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" event={"ID":"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf","Type":"ContainerDied","Data":"bb047c1ffd10df71ec667eaacfdc46766e3f8b2a469d70cc27c5d32aec157204"} Oct 30 01:17:14.882442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.882317 5055 scope.go:122] "RemoveContainer" containerID="d3e27a8324376e2d5c186fcfa676d1a7925a6e551c97b7f07801c298375c6438" Oct 30 01:17:14.882810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.882795 5055 scope.go:122] "RemoveContainer" containerID="bb047c1ffd10df71ec667eaacfdc46766e3f8b2a469d70cc27c5d32aec157204" Oct 30 01:17:14.883042 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:14.883019 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tw27b_openshift-console-operator(b6cb0d17-b098-408e-92b6-b5e81a8bb9cf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" podUID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" Oct 30 01:17:14.885949 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.885886 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdpcp" event={"ID":"1e24a38d-9f35-40bf-bb90-57fee63d8ac0","Type":"ContainerStarted","Data":"0a42a15d22bbadaf051e3e69842abb8e3262565d403da64c24b059936a7e874f"} Oct 30 01:17:14.887589 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.887539 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" event={"ID":"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41","Type":"ContainerStarted","Data":"71fc392ed0373fef61066a251b197c41a5b0b12629832cdaf8b3d590c87d8260"} Oct 30 01:17:14.889772 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.889746 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" event={"ID":"806fd3b6-00de-4f42-8c35-e638d3835ec7","Type":"ContainerStarted","Data":"2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb"} Oct 30 01:17:14.889877 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.889792 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" event={"ID":"806fd3b6-00de-4f42-8c35-e638d3835ec7","Type":"ContainerStarted","Data":"25be08911bf0ee43bd80e546c4a59e397e8fe230dfcbc2d7647418f28dec9cff"} Oct 30 01:17:14.891497 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.891437 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" event={"ID":"4588e6fe-4b1e-4e62-8528-7411e598b444","Type":"ContainerStarted","Data":"676473ff69a51db34d224a97f54ab431ab40a644bac89b3b1cd47c227ea8adf9"} Oct 30 01:17:14.893007 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.892983 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9qrg" event={"ID":"d8c54d89-eea4-45ee-80e5-fd0aba53e274","Type":"ContainerStarted","Data":"da523ff28ab8c1b92da2471fe5d8d70786c98099bb2f76089fc6540a28ce015c"} Oct 30 01:17:14.896060 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.895985 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655b484568-p27mb" event={"ID":"7082f68a-2079-426e-997f-1d3ba3997ba6","Type":"ContainerStarted","Data":"525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55"} Oct 30 01:17:14.896060 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:14.896033 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655b484568-p27mb" event={"ID":"7082f68a-2079-426e-997f-1d3ba3997ba6","Type":"ContainerStarted","Data":"a879ec3a72661efac01ed9379c5a826dae53540bb3c8e2ee36adfe094cffa0c0"} Oct 30 01:17:15.091870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.091838 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lh2vb_8855c2e7-9182-4ae6-b96a-50baa69d9a67/serve-healthcheck-canary/0.log" Oct 30 01:17:15.290391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.290361 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-jgj26_cc0b4765-071a-4af8-88e6-700abd1d8f83/migrator/0.log" Oct 30 01:17:15.490392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.490363 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-jj7dm_56fd7c23-5b71-424f-87e6-91f34918efe8/migrator/0.log" Oct 30 01:17:15.564392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.563458 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:15.564392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.563577 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:15.564392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.563610 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:15.564392 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:15.563760 5055 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:17:15.564392 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:15.563830 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:17.563812409 +0000 UTC m=+4.884769434 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : secret "insights-runtime-extractor-tls" not found Oct 30 01:17:15.564392 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:15.563904 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:17.56389351 +0000 UTC m=+4.884850532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : configmap references non-existent config key: service-ca.crt Oct 30 01:17:15.564392 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:15.563962 5055 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:15.564392 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:15.563992 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls podName:c9b752e3-e38a-4d0f-babd-7922d6a2e872 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:17.563983103 +0000 UTC m=+4.884940123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pb877" (UID: "c9b752e3-e38a-4d0f-babd-7922d6a2e872") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:15.691587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.691485 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v_bb940654-4320-4340-a2d5-9450bc68f40f/kube-storage-version-migrator-operator/0.log" Oct 30 01:17:15.695012 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.694986 5055 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:15.901768 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.901719 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:17:15.902086 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.901827 5055 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:17:15.902243 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.902228 5055 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:17:15.902300 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:15.902275 5055 scope.go:122] "RemoveContainer" containerID="bb047c1ffd10df71ec667eaacfdc46766e3f8b2a469d70cc27c5d32aec157204" Oct 30 01:17:15.902530 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:15.902501 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tw27b_openshift-console-operator(b6cb0d17-b098-408e-92b6-b5e81a8bb9cf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" podUID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" Oct 30 01:17:16.906872 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.906842 5055 generic.go:356] "Generic (PLEG): container finished" podID="bb940654-4320-4340-a2d5-9450bc68f40f" containerID="674cb174aefe2d90a87f442845e6021a0c7dccaf30de5c8b7871a5320eb3b494" exitCode=0 Oct 30 01:17:16.907254 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.906922 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" event={"ID":"bb940654-4320-4340-a2d5-9450bc68f40f","Type":"ContainerDied","Data":"674cb174aefe2d90a87f442845e6021a0c7dccaf30de5c8b7871a5320eb3b494"} Oct 30 01:17:16.907353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.907310 5055 scope.go:122] "RemoveContainer" containerID="674cb174aefe2d90a87f442845e6021a0c7dccaf30de5c8b7871a5320eb3b494" Oct 30 01:17:16.908484 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.908465 5055 generic.go:356] "Generic (PLEG): container finished" podID="712bffb7-97f2-422a-ba7b-cce9eafccba4" containerID="51d87176a0f58d0c89903dad4b40b47490319828a6787d6294139d0e2887c900" exitCode=0 Oct 30 01:17:16.908575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.908487 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" event={"ID":"712bffb7-97f2-422a-ba7b-cce9eafccba4","Type":"ContainerDied","Data":"51d87176a0f58d0c89903dad4b40b47490319828a6787d6294139d0e2887c900"} Oct 30 01:17:16.908808 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.908786 5055 scope.go:122] "RemoveContainer" containerID="51d87176a0f58d0c89903dad4b40b47490319828a6787d6294139d0e2887c900" Oct 30 01:17:16.910315 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.910293 5055 generic.go:356] "Generic (PLEG): container finished" podID="083c7373-177e-46cb-99c1-f9509121668a" containerID="c02f7ece88e3f27ea8d6f5775cc36faa3c688969f91108c83f3f95744c7ddea3" exitCode=0 Oct 30 01:17:16.910397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.910362 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-64j9l" event={"ID":"083c7373-177e-46cb-99c1-f9509121668a","Type":"ContainerDied","Data":"c02f7ece88e3f27ea8d6f5775cc36faa3c688969f91108c83f3f95744c7ddea3"} Oct 30 01:17:16.910677 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.910660 5055 scope.go:122] "RemoveContainer" containerID="c02f7ece88e3f27ea8d6f5775cc36faa3c688969f91108c83f3f95744c7ddea3" Oct 30 01:17:16.918721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.918682 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdpcp" event={"ID":"1e24a38d-9f35-40bf-bb90-57fee63d8ac0","Type":"ContainerStarted","Data":"292eba158f0828e3f1ea9d2d8faf5c3a1e04a5ae033b60bc1eecc1585176131d"} Oct 30 01:17:16.918823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.918726 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qdpcp" event={"ID":"1e24a38d-9f35-40bf-bb90-57fee63d8ac0","Type":"ContainerStarted","Data":"89112b7feac398e0a84d784d385fac66e7dfefedb00759e5bea20b102d8448c4"} Oct 30 01:17:16.918823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.918784 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:16.922020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.921997 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-kzdph" event={"ID":"0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41","Type":"ContainerStarted","Data":"84bade78ee74f222e6149567abcb82c2c2660296e750ce33120b548d08cb2592"} Oct 30 01:17:16.924116 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.924091 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" event={"ID":"4588e6fe-4b1e-4e62-8528-7411e598b444","Type":"ContainerStarted","Data":"29faf52fc845fe0032c9e27749cbcd9918ce86792c623ea648caa773d4d30975"} Oct 30 01:17:16.924202 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.924137 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-54ddb" event={"ID":"4588e6fe-4b1e-4e62-8528-7411e598b444","Type":"ContainerStarted","Data":"47327317c9eb2054a82e4168e33911ddf4463b78c7d0c1294db9493cc63c67a5"} Oct 30 01:17:16.926261 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.926237 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9qrg" event={"ID":"d8c54d89-eea4-45ee-80e5-fd0aba53e274","Type":"ContainerStarted","Data":"0cfdda9e2ce9d1f5cb3938b9ab5f11bbf4cf710a8d9acea37030e24ec63f5a77"} Oct 30 01:17:16.926368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:16.926283 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p9qrg" event={"ID":"d8c54d89-eea4-45ee-80e5-fd0aba53e274","Type":"ContainerStarted","Data":"215f2d0eafcbf295d4b176122d25c773e89ff78d561e3d588466c0eb65ff2d9a"} Oct 30 01:17:17.586200 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:17.586163 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:17.586384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:17.586242 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:17.586384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:17.586267 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:17.586384 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:17.586368 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle podName:a310cb37-d20b-4b25-9796-c5f56c44adac nodeName:}" failed. No retries permitted until 2026-10-30 01:17:21.58634931 +0000 UTC m=+8.907306340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle") pod "router-default-67bf8c5c5-mtkk5" (UID: "a310cb37-d20b-4b25-9796-c5f56c44adac") : configmap references non-existent config key: service-ca.crt Oct 30 01:17:17.586532 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:17.586400 5055 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:17.586532 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:17.586450 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls podName:c9b752e3-e38a-4d0f-babd-7922d6a2e872 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:21.5864397 +0000 UTC m=+8.907396735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-pb877" (UID: "c9b752e3-e38a-4d0f-babd-7922d6a2e872") : secret "cluster-monitoring-operator-tls" not found Oct 30 01:17:17.586532 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:17.586399 5055 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 01:17:17.586532 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:17.586489 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls podName:184de02f-0273-408d-9ee5-265a0bb278a4 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:21.58648008 +0000 UTC m=+8.907437113 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jz2gt" (UID: "184de02f-0273-408d-9ee5-265a0bb278a4") : secret "insights-runtime-extractor-tls" not found Oct 30 01:17:17.918709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:17.918635 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:17.930873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:17.930845 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-64j9l" event={"ID":"083c7373-177e-46cb-99c1-f9509121668a","Type":"ContainerStarted","Data":"312558720a6fa2b7da581f7353b0b1f5a7068495bd84fd3dfd3e6113492995c5"} Oct 30 01:17:17.932742 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:17.932721 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-2wj6v" event={"ID":"bb940654-4320-4340-a2d5-9450bc68f40f","Type":"ContainerStarted","Data":"305d181bec1981a456983307de5398f0119913206e4ffcc7048ac9946c2475b4"} Oct 30 01:17:17.934526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:17.934503 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-tm7mf" event={"ID":"712bffb7-97f2-422a-ba7b-cce9eafccba4","Type":"ContainerStarted","Data":"9f85415a2a83c18de5700e3a31f3a2e37cfa178854a188447b078f9982458c30"} Oct 30 01:17:19.270342 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:19.270293 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:19.270732 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:19.270542 5055 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:17:19.759265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:19.759237 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:21.619443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.619396 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:21.619443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.619448 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:21.619890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.619490 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:21.620076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.620042 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a310cb37-d20b-4b25-9796-c5f56c44adac-service-ca-bundle\") pod \"router-default-67bf8c5c5-mtkk5\" (UID: \"a310cb37-d20b-4b25-9796-c5f56c44adac\") " pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:21.622086 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.622060 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/184de02f-0273-408d-9ee5-265a0bb278a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jz2gt\" (UID: \"184de02f-0273-408d-9ee5-265a0bb278a4\") " pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:21.622148 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.622120 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/c9b752e3-e38a-4d0f-babd-7922d6a2e872-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-pb877\" (UID: \"c9b752e3-e38a-4d0f-babd-7922d6a2e872\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:21.629941 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.629919 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jz2gt" Oct 30 01:17:21.914907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.914837 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:21.921404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.921192 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" Oct 30 01:17:21.924590 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.924568 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jz2gt"] Oct 30 01:17:21.929125 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:21.929097 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184de02f_0273_408d_9ee5_265a0bb278a4.slice/crio-47cdd42ad380ffbb642bb3cdbfd304e36979487fadb4e48356016e9ddb101431 WatchSource:0}: Error finding container 47cdd42ad380ffbb642bb3cdbfd304e36979487fadb4e48356016e9ddb101431: Status 404 returned error can't find the container with id 47cdd42ad380ffbb642bb3cdbfd304e36979487fadb4e48356016e9ddb101431 Oct 30 01:17:21.954538 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:21.954498 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jz2gt" event={"ID":"184de02f-0273-408d-9ee5-265a0bb278a4","Type":"ContainerStarted","Data":"47cdd42ad380ffbb642bb3cdbfd304e36979487fadb4e48356016e9ddb101431"} Oct 30 01:17:22.046057 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:22.046029 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:22.046500 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:22.046482 5055 scope.go:122] "RemoveContainer" containerID="bb047c1ffd10df71ec667eaacfdc46766e3f8b2a469d70cc27c5d32aec157204" Oct 30 01:17:22.046713 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:22.046693 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-tw27b_openshift-console-operator(b6cb0d17-b098-408e-92b6-b5e81a8bb9cf)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" podUID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" Oct 30 01:17:22.231748 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:22.231722 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-pb877"] Oct 30 01:17:22.232505 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:22.232481 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b752e3_e38a_4d0f_babd_7922d6a2e872.slice/crio-ae0c943b0e7e98d512d196c3e3e9e0600bced6e955e68050722f49a7a441daf8 WatchSource:0}: Error finding container ae0c943b0e7e98d512d196c3e3e9e0600bced6e955e68050722f49a7a441daf8: Status 404 returned error can't find the container with id ae0c943b0e7e98d512d196c3e3e9e0600bced6e955e68050722f49a7a441daf8 Oct 30 01:17:22.234593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:22.234574 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-67bf8c5c5-mtkk5"] Oct 30 01:17:22.235133 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:22.235109 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda310cb37_d20b_4b25_9796_c5f56c44adac.slice/crio-fd17c8c998006935415cbf413c88e13bb0c7fa7a6a8ed74c53cd2963841bb902 WatchSource:0}: Error finding container fd17c8c998006935415cbf413c88e13bb0c7fa7a6a8ed74c53cd2963841bb902: Status 404 returned error can't find the container with id fd17c8c998006935415cbf413c88e13bb0c7fa7a6a8ed74c53cd2963841bb902 Oct 30 01:17:22.960699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:22.960654 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" event={"ID":"a310cb37-d20b-4b25-9796-c5f56c44adac","Type":"ContainerStarted","Data":"31b6fbf511fb28fe8ed0715c4d640d6fe5a26d47447cf428b3e39be7f86bd778"} Oct 30 01:17:22.961137 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:22.960717 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" event={"ID":"a310cb37-d20b-4b25-9796-c5f56c44adac","Type":"ContainerStarted","Data":"fd17c8c998006935415cbf413c88e13bb0c7fa7a6a8ed74c53cd2963841bb902"} Oct 30 01:17:22.961937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:22.961880 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" event={"ID":"c9b752e3-e38a-4d0f-babd-7922d6a2e872","Type":"ContainerStarted","Data":"ae0c943b0e7e98d512d196c3e3e9e0600bced6e955e68050722f49a7a441daf8"} Oct 30 01:17:22.963570 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:22.963545 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jz2gt" event={"ID":"184de02f-0273-408d-9ee5-265a0bb278a4","Type":"ContainerStarted","Data":"304ab1884a07a2ef80af87fce63d1ab8e4b50c1f6c31d06d460da885fa4ad991"} Oct 30 01:17:22.963653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:22.963601 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jz2gt" event={"ID":"184de02f-0273-408d-9ee5-265a0bb278a4","Type":"ContainerStarted","Data":"a24f1c6c369baee3b13da6798b0d2c770764bb4a7cc8b2e4f92db7ac0343b0d2"} Oct 30 01:17:23.399296 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:23.399215 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:23.915126 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:23.915096 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:23.917827 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:23.917806 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:23.971885 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:23.971849 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-pb877" event={"ID":"c9b752e3-e38a-4d0f-babd-7922d6a2e872","Type":"ContainerStarted","Data":"1cfbe999757948a5e5dc894ec9e1d279aedaa09522f51f9ae7931166b844fc82"} Oct 30 01:17:24.818450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:24.818380 5055 scope.go:122] "RemoveContainer" containerID="bb047c1ffd10df71ec667eaacfdc46766e3f8b2a469d70cc27c5d32aec157204" Oct 30 01:17:24.976284 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:24.976260 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:17:24.976714 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:24.976356 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" event={"ID":"b6cb0d17-b098-408e-92b6-b5e81a8bb9cf","Type":"ContainerStarted","Data":"167bf40bcf672f859d753835614f968dbb544747711c69a2991479333e4488c6"} Oct 30 01:17:24.976714 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:24.976668 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:24.977849 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:24.977823 5055 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-tw27b container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.6:8443/readyz\": dial tcp 10.132.0.6:8443: connect: connection refused" start-of-body= Oct 30 01:17:24.977964 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:24.977867 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" podUID="b6cb0d17-b098-408e-92b6-b5e81a8bb9cf" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.6:8443/readyz\": dial tcp 10.132.0.6:8443: connect: connection refused" Oct 30 01:17:24.978787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:24.978757 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jz2gt" event={"ID":"184de02f-0273-408d-9ee5-265a0bb278a4","Type":"ContainerStarted","Data":"5eba95fd9275f3df0eb6c35cf0c1dab00570d4855d4650792d42a606818a1fca"} Oct 30 01:17:24.978921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:24.978905 5055 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:17:25.987912 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:25.987882 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-tw27b" Oct 30 01:17:26.174818 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:26.174789 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mvmkl" Oct 30 01:17:26.676657 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:26.676619 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vx2b8" Oct 30 01:17:26.937785 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:26.937711 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qdpcp" Oct 30 01:17:27.216077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:27.216049 5055 patch_prober.go:29] interesting pod/image-registry-54db9d5677-ptwtb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 01:17:27.216462 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:27.216101 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" podUID="806fd3b6-00de-4f42-8c35-e638d3835ec7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:17:27.547742 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:27.547656 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:27.547910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:27.547784 5055 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:17:27.549083 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:27.549052 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-67bf8c5c5-mtkk5" Oct 30 01:17:28.304365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.301169 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rwmdr-6hm72"] Oct 30 01:17:28.304754 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.304680 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.333974 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.333944 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt"] Oct 30 01:17:28.336461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.336439 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt" Oct 30 01:17:28.369519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.369492 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rwmdr-6hm72"] Oct 30 01:17:28.372892 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.372866 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 01:17:28.374014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.373987 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt"] Oct 30 01:17:28.379289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.379245 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-trusted-ca-bundle\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.379426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.379318 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2592437b-af46-4f92-8db7-0580d36f7f80-archives-path\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.379426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.379377 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swzmk\" (UniqueName: \"kubernetes.io/projected/2592437b-af46-4f92-8db7-0580d36f7f80-kube-api-access-swzmk\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.379426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.379406 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6a6e53e8-68df-4699-b745-d2183fb8d473-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6lhwt\" (UID: \"6a6e53e8-68df-4699-b745-d2183fb8d473\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt" Oct 30 01:17:28.379523 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.379469 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-service-ca-bundle\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.382365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.381583 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6wk8z\"" Oct 30 01:17:28.404149 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.404118 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-655b484568-p27mb"] Oct 30 01:17:28.424443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.415537 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-nnzpp"] Oct 30 01:17:28.424443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.422546 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nnzpp" Oct 30 01:17:28.445499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.445440 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 01:17:28.445706 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.445691 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 01:17:28.450669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.450641 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nnzpp"] Oct 30 01:17:28.478365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.475629 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-zwjgn\"" Oct 30 01:17:28.482350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.480015 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkrl8\" (UniqueName: \"kubernetes.io/projected/0855692d-c388-4913-a361-7af6c41bac7a-kube-api-access-mkrl8\") pod \"downloads-6dd789c8b7-nnzpp\" (UID: \"0855692d-c388-4913-a361-7af6c41bac7a\") " pod="openshift-console/downloads-6dd789c8b7-nnzpp" Oct 30 01:17:28.482350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.480070 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6a6e53e8-68df-4699-b745-d2183fb8d473-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6lhwt\" (UID: \"6a6e53e8-68df-4699-b745-d2183fb8d473\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt" Oct 30 01:17:28.482350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.480133 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-service-ca-bundle\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.482350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.480171 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-trusted-ca-bundle\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.482350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.480220 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0855692d-c388-4913-a361-7af6c41bac7a-tmp\") pod \"downloads-6dd789c8b7-nnzpp\" (UID: \"0855692d-c388-4913-a361-7af6c41bac7a\") " pod="openshift-console/downloads-6dd789c8b7-nnzpp" Oct 30 01:17:28.482350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.480246 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2592437b-af46-4f92-8db7-0580d36f7f80-archives-path\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.482350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.480285 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swzmk\" (UniqueName: \"kubernetes.io/projected/2592437b-af46-4f92-8db7-0580d36f7f80-kube-api-access-swzmk\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.482756 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.482384 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-trusted-ca-bundle\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.484732 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.482968 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-service-ca-bundle\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.484843 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.484799 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2592437b-af46-4f92-8db7-0580d36f7f80-archives-path\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.486964 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.486945 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6a6e53e8-68df-4699-b745-d2183fb8d473-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6lhwt\" (UID: \"6a6e53e8-68df-4699-b745-d2183fb8d473\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt" Oct 30 01:17:28.508975 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.508946 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d99f44fbc-h76gr"] Oct 30 01:17:28.511658 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.511634 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swzmk\" (UniqueName: \"kubernetes.io/projected/2592437b-af46-4f92-8db7-0580d36f7f80-kube-api-access-swzmk\") pod \"periodic-gathering-rwmdr-6hm72\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.512282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.512264 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.541542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.541520 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d99f44fbc-h76gr"] Oct 30 01:17:28.590137 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590067 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-registry-certificates\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.590137 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590110 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-installation-pull-secrets\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.590458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590144 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-bound-sa-token\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.590458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590185 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-image-registry-private-configuration\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.590458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590237 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d22f\" (UniqueName: \"kubernetes.io/projected/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-kube-api-access-2d22f\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.590458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590278 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0855692d-c388-4913-a361-7af6c41bac7a-tmp\") pod \"downloads-6dd789c8b7-nnzpp\" (UID: \"0855692d-c388-4913-a361-7af6c41bac7a\") " pod="openshift-console/downloads-6dd789c8b7-nnzpp" Oct 30 01:17:28.590458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590302 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-registry-tls\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.590458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590367 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-trusted-ca\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.590458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590395 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkrl8\" (UniqueName: \"kubernetes.io/projected/0855692d-c388-4913-a361-7af6c41bac7a-kube-api-access-mkrl8\") pod \"downloads-6dd789c8b7-nnzpp\" (UID: \"0855692d-c388-4913-a361-7af6c41bac7a\") " pod="openshift-console/downloads-6dd789c8b7-nnzpp" Oct 30 01:17:28.590458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590419 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-ca-trust-extracted\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.590911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.590889 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0855692d-c388-4913-a361-7af6c41bac7a-tmp\") pod \"downloads-6dd789c8b7-nnzpp\" (UID: \"0855692d-c388-4913-a361-7af6c41bac7a\") " pod="openshift-console/downloads-6dd789c8b7-nnzpp" Oct 30 01:17:28.616821 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.616790 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:17:28.619936 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.619907 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkrl8\" (UniqueName: \"kubernetes.io/projected/0855692d-c388-4913-a361-7af6c41bac7a-kube-api-access-mkrl8\") pod \"downloads-6dd789c8b7-nnzpp\" (UID: \"0855692d-c388-4913-a361-7af6c41bac7a\") " pod="openshift-console/downloads-6dd789c8b7-nnzpp" Oct 30 01:17:28.657461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.657431 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt" Oct 30 01:17:28.690967 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.690938 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-bound-sa-token\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.691095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.690988 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-image-registry-private-configuration\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.691095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.691035 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d22f\" (UniqueName: \"kubernetes.io/projected/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-kube-api-access-2d22f\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.691095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.691073 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-registry-tls\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.691252 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.691110 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-trusted-ca\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.691252 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.691132 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-ca-trust-extracted\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.691252 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.691182 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-registry-certificates\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.691252 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.691205 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-installation-pull-secrets\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.694112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.692584 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-registry-certificates\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.694112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.693663 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-ca-trust-extracted\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.694418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.694395 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-trusted-ca\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.712022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.711985 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-image-registry-private-configuration\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.718438 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.717859 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-installation-pull-secrets\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.718438 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.718256 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-registry-tls\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.724363 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.724263 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-bound-sa-token\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.724629 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.724610 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d22f\" (UniqueName: \"kubernetes.io/projected/26443b6e-20d3-47e9-b0fd-e21d38b42e7c-kube-api-access-2d22f\") pod \"image-registry-6d99f44fbc-h76gr\" (UID: \"26443b6e-20d3-47e9-b0fd-e21d38b42e7c\") " pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.739432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.738403 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-nnzpp" Oct 30 01:17:28.845522 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.845474 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:28.953537 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.953406 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rwmdr-6hm72"] Oct 30 01:17:28.989087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:28.989061 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt"] Oct 30 01:17:28.989515 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:28.989485 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6e53e8_68df_4699_b745_d2183fb8d473.slice/crio-07fecc4bebd11a5580472f050d9263481c704433ee2b46a5b947ecf1ba8435a8 WatchSource:0}: Error finding container 07fecc4bebd11a5580472f050d9263481c704433ee2b46a5b947ecf1ba8435a8: Status 404 returned error can't find the container with id 07fecc4bebd11a5580472f050d9263481c704433ee2b46a5b947ecf1ba8435a8 Oct 30 01:17:29.003709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:29.001178 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rwmdr-6hm72" event={"ID":"2592437b-af46-4f92-8db7-0580d36f7f80","Type":"ContainerStarted","Data":"4806a3aeca46022a6e9dc006e668223c21991080f1229eff6325b825e4a82d7e"} Oct 30 01:17:29.003709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:29.002721 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt" event={"ID":"6a6e53e8-68df-4699-b745-d2183fb8d473","Type":"ContainerStarted","Data":"07fecc4bebd11a5580472f050d9263481c704433ee2b46a5b947ecf1ba8435a8"} Oct 30 01:17:29.090696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:29.089806 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-nnzpp"] Oct 30 01:17:29.090696 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:29.089903 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0855692d_c388_4913_a361_7af6c41bac7a.slice/crio-1ac987e4dd5c57a423e7d9579362279cd684b78a694eedda9a2e879859f1853a WatchSource:0}: Error finding container 1ac987e4dd5c57a423e7d9579362279cd684b78a694eedda9a2e879859f1853a: Status 404 returned error can't find the container with id 1ac987e4dd5c57a423e7d9579362279cd684b78a694eedda9a2e879859f1853a Oct 30 01:17:29.167389 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:29.167360 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d99f44fbc-h76gr"] Oct 30 01:17:29.167959 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:29.167935 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26443b6e_20d3_47e9_b0fd_e21d38b42e7c.slice/crio-5cba5f74dc0935cc74e5a5321accf0eb4b1cbbd5e9675eab2dbbea22de2ccb57 WatchSource:0}: Error finding container 5cba5f74dc0935cc74e5a5321accf0eb4b1cbbd5e9675eab2dbbea22de2ccb57: Status 404 returned error can't find the container with id 5cba5f74dc0935cc74e5a5321accf0eb4b1cbbd5e9675eab2dbbea22de2ccb57 Oct 30 01:17:29.765426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:29.765391 5055 patch_prober.go:29] interesting pod/image-registry-54db9d5677-ptwtb container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 01:17:29.765920 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:29.765448 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" podUID="806fd3b6-00de-4f42-8c35-e638d3835ec7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:17:30.012002 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:30.011226 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rwmdr-6hm72" event={"ID":"2592437b-af46-4f92-8db7-0580d36f7f80","Type":"ContainerStarted","Data":"43c2ce6c6b7f59adc83e444dc98f7ca50a8b231d02014a69f1b9a9af7e7f242f"} Oct 30 01:17:30.015800 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:30.015731 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" event={"ID":"26443b6e-20d3-47e9-b0fd-e21d38b42e7c","Type":"ContainerStarted","Data":"659bc2948ee61fb5429cab9d8d0b70be489244168f06807c0c90f63c34449a7c"} Oct 30 01:17:30.015800 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:30.015788 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:30.015974 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:30.015810 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" event={"ID":"26443b6e-20d3-47e9-b0fd-e21d38b42e7c","Type":"ContainerStarted","Data":"5cba5f74dc0935cc74e5a5321accf0eb4b1cbbd5e9675eab2dbbea22de2ccb57"} Oct 30 01:17:30.017292 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:30.017261 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nnzpp" event={"ID":"0855692d-c388-4913-a361-7af6c41bac7a","Type":"ContainerStarted","Data":"1ac987e4dd5c57a423e7d9579362279cd684b78a694eedda9a2e879859f1853a"} Oct 30 01:17:30.053788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:30.053720 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rwmdr-6hm72" podStartSLOduration=2.053699929 podStartE2EDuration="2.053699929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:30.052239785 +0000 UTC m=+17.373196826" watchObservedRunningTime="2026-10-30 01:17:30.053699929 +0000 UTC m=+17.374656971" Oct 30 01:17:30.095458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:30.095402 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" podStartSLOduration=2.095384579 podStartE2EDuration="2.095384579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:30.09464512 +0000 UTC m=+17.415602161" watchObservedRunningTime="2026-10-30 01:17:30.095384579 +0000 UTC m=+17.416341622" Oct 30 01:17:30.820435 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:30.820407 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdpcp_1e24a38d-9f35-40bf-bb90-57fee63d8ac0/dns/0.log" Oct 30 01:17:31.005378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:31.005352 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdpcp_1e24a38d-9f35-40bf-bb90-57fee63d8ac0/kube-rbac-proxy/0.log" Oct 30 01:17:31.023614 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:31.023583 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt" event={"ID":"6a6e53e8-68df-4699-b745-d2183fb8d473","Type":"ContainerStarted","Data":"e1c29f6706bd70094c5985d4d6bc36d515e2188855470eec85ae14b387f8ff86"} Oct 30 01:17:31.069465 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:31.069424 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt" podStartSLOduration=1.951837112 podStartE2EDuration="3.069411099s" totalImagesPullingTime="1.117573987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:31.06843942 +0000 UTC m=+18.389396460" watchObservedRunningTime="2026-10-30 01:17:31.069411099 +0000 UTC m=+18.390368139" Oct 30 01:17:31.410166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:31.410135 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9gckm_b47f8e4a-5060-47bf-8bec-722ccff0fa51/dns-node-resolver/0.log" Oct 30 01:17:31.806791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:31.806759 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-54db9d5677-ptwtb_806fd3b6-00de-4f42-8c35-e638d3835ec7/registry/0.log" Oct 30 01:17:32.007212 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:32.007183 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-655b484568-p27mb_7082f68a-2079-426e-997f-1d3ba3997ba6/registry/0.log" Oct 30 01:17:32.030257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:32.030226 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt" Oct 30 01:17:32.035874 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:32.035851 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6lhwt" Oct 30 01:17:32.205361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:32.205310 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6d99f44fbc-h76gr_26443b6e-20d3-47e9-b0fd-e21d38b42e7c/registry/0.log" Oct 30 01:17:32.612776 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:32.612654 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kckpv_c9801105-8f8a-45d0-b2c1-23468a212347/node-ca/0.log" Oct 30 01:17:33.006210 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:33.006185 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67bf8c5c5-mtkk5_a310cb37-d20b-4b25-9796-c5f56c44adac/router/0.log" Oct 30 01:17:33.406150 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:33.406064 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lh2vb_8855c2e7-9182-4ae6-b96a-50baa69d9a67/serve-healthcheck-canary/0.log" Oct 30 01:17:37.548838 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.548806 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vc8rt"] Oct 30 01:17:37.584772 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.584738 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.620558 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.620457 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 01:17:37.621015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.620974 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 01:17:37.621144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.621058 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 01:17:37.621219 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.621161 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vthnr\"" Oct 30 01:17:37.621270 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.621244 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 01:17:37.678736 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.678697 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-accelerators-collector-config\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.678888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.678791 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-sys\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.678888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.678841 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-textfile\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.678974 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.678900 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bl4n\" (UniqueName: \"kubernetes.io/projected/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-kube-api-access-8bl4n\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.678974 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.678931 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-root\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.679045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.678972 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-tls\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.679045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.678999 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-wtmp\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.679045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.679019 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-metrics-client-ca\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.679179 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.679044 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780206 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780169 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-tls\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780373 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780215 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-wtmp\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780430 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780373 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-wtmp\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780430 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780385 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-metrics-client-ca\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780504 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780430 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780504 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780483 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-accelerators-collector-config\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780563 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780525 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-sys\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780558 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-textfile\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780667 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780633 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bl4n\" (UniqueName: \"kubernetes.io/projected/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-kube-api-access-8bl4n\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780670 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-root\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780769 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780750 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-sys\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.780769 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.780759 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-root\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.781036 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.781012 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-metrics-client-ca\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.781457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.781430 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-textfile\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.781578 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.781556 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-accelerators-collector-config\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.783314 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.783294 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-tls\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.783614 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.783593 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.802180 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.802124 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bl4n\" (UniqueName: \"kubernetes.io/projected/907bc83e-74a4-4edb-8cf3-28d0fd1a6963-kube-api-access-8bl4n\") pod \"node-exporter-vc8rt\" (UID: \"907bc83e-74a4-4edb-8cf3-28d0fd1a6963\") " pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.952782 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:37.952755 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vc8rt" Oct 30 01:17:37.986144 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:17:37.986112 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907bc83e_74a4_4edb_8cf3_28d0fd1a6963.slice/crio-943e7f7bc4994ba96eb762b2d61404eb0f1ba765a70644b419abd57d474af1d3 WatchSource:0}: Error finding container 943e7f7bc4994ba96eb762b2d61404eb0f1ba765a70644b419abd57d474af1d3: Status 404 returned error can't find the container with id 943e7f7bc4994ba96eb762b2d61404eb0f1ba765a70644b419abd57d474af1d3 Oct 30 01:17:38.061040 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:38.060965 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vc8rt" event={"ID":"907bc83e-74a4-4edb-8cf3-28d0fd1a6963","Type":"ContainerStarted","Data":"943e7f7bc4994ba96eb762b2d61404eb0f1ba765a70644b419abd57d474af1d3"} Oct 30 01:17:38.410598 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:38.410510 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:39.764602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:39.764537 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:17:40.074163 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:40.074071 5055 generic.go:356] "Generic (PLEG): container finished" podID="907bc83e-74a4-4edb-8cf3-28d0fd1a6963" containerID="e486b509baf4385da49cfd770bed0e6d25c36eb39f5feb7ac1dfa32eec013048" exitCode=0 Oct 30 01:17:40.074163 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:40.074145 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vc8rt" event={"ID":"907bc83e-74a4-4edb-8cf3-28d0fd1a6963","Type":"ContainerDied","Data":"e486b509baf4385da49cfd770bed0e6d25c36eb39f5feb7ac1dfa32eec013048"} Oct 30 01:17:41.082054 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:41.081961 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vc8rt" event={"ID":"907bc83e-74a4-4edb-8cf3-28d0fd1a6963","Type":"ContainerStarted","Data":"c1dd69d7914212fd527604d4ea9df08e5ff0718cd251169237863731f8b3618e"} Oct 30 01:17:41.082054 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:41.082023 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vc8rt" event={"ID":"907bc83e-74a4-4edb-8cf3-28d0fd1a6963","Type":"ContainerStarted","Data":"74b3fd7dd0eef018ac23ed226608fba39bf3fca55194d3411b113a9281a427af"} Oct 30 01:17:41.144659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:41.144594 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vc8rt" podStartSLOduration=2.5563001180000002 podStartE2EDuration="4.144576768s" totalImagesPullingTime="1.48418026s" totalInitContainerRuntime="104.09639ms" isStatefulPod=true podCreationTimestamp="2026-10-30 01:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:41.144434127 +0000 UTC m=+28.465391168" watchObservedRunningTime="2026-10-30 01:17:41.144576768 +0000 UTC m=+28.465533811" Oct 30 01:17:44.899106 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:44.899072 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4cnv8" Oct 30 01:17:46.107737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:46.107702 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-nnzpp" event={"ID":"0855692d-c388-4913-a361-7af6c41bac7a","Type":"ContainerStarted","Data":"22fd1d3c6d2ba03be1f9c8bca87523b022be73933abc7fb68dd3ea1e93af5cc6"} Oct 30 01:17:46.108221 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:46.108061 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-nnzpp" Oct 30 01:17:46.122508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:46.122480 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-nnzpp" Oct 30 01:17:46.176560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:46.176505 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-nnzpp" podStartSLOduration=1.764584752 podStartE2EDuration="18.176492415s" totalImagesPullingTime="16.411907663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:46.163304634 +0000 UTC m=+33.484261673" watchObservedRunningTime="2026-10-30 01:17:46.176492415 +0000 UTC m=+33.497449476" Oct 30 01:17:49.222658 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:49.222611 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:17:51.030378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:51.030352 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d99f44fbc-h76gr" Oct 30 01:17:51.144767 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:51.144736 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54db9d5677-ptwtb"] Oct 30 01:17:53.433027 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.432911 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-655b484568-p27mb" podUID="7082f68a-2079-426e-997f-1d3ba3997ba6" containerName="registry" containerID="cri-o://525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55" gracePeriod=30 Oct 30 01:17:53.740119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.740080 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:53.845290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.845258 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-certificates\") pod \"7082f68a-2079-426e-997f-1d3ba3997ba6\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " Oct 30 01:17:53.845481 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.845359 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-kube-api-access-46q9r\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-kube-api-access-46q9r\") pod \"7082f68a-2079-426e-997f-1d3ba3997ba6\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " Oct 30 01:17:53.845481 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.845403 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-trusted-ca\") pod \"7082f68a-2079-426e-997f-1d3ba3997ba6\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " Oct 30 01:17:53.845481 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.845441 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-installation-pull-secrets\") pod \"7082f68a-2079-426e-997f-1d3ba3997ba6\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " Oct 30 01:17:53.845669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.845489 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-image-registry-private-configuration\") pod \"7082f68a-2079-426e-997f-1d3ba3997ba6\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " Oct 30 01:17:53.845669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.845546 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-bound-sa-token\") pod \"7082f68a-2079-426e-997f-1d3ba3997ba6\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " Oct 30 01:17:53.845669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.845579 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") pod \"7082f68a-2079-426e-997f-1d3ba3997ba6\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " Oct 30 01:17:53.845669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.845611 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7082f68a-2079-426e-997f-1d3ba3997ba6-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7082f68a-2079-426e-997f-1d3ba3997ba6-ca-trust-extracted\") pod \"7082f68a-2079-426e-997f-1d3ba3997ba6\" (UID: \"7082f68a-2079-426e-997f-1d3ba3997ba6\") " Oct 30 01:17:53.845868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.845737 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-certificates" pod "7082f68a-2079-426e-997f-1d3ba3997ba6" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:53.845919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.845890 5055 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-certificates\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:17:53.846421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.846390 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-trusted-ca" pod "7082f68a-2079-426e-997f-1d3ba3997ba6" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:17:53.850165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.848997 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-kube-api-access-46q9r" pod "7082f68a-2079-426e-997f-1d3ba3997ba6" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6"). InnerVolumeSpecName "kube-api-access-46q9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:53.850165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.849247 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-image-registry-private-configuration" pod "7082f68a-2079-426e-997f-1d3ba3997ba6" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:53.855499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.855471 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7082f68a-2079-426e-997f-1d3ba3997ba6-ca-trust-extracted" pod "7082f68a-2079-426e-997f-1d3ba3997ba6" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:17:53.855878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.855842 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls" pod "7082f68a-2079-426e-997f-1d3ba3997ba6" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:53.855991 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.855877 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-installation-pull-secrets" pod "7082f68a-2079-426e-997f-1d3ba3997ba6" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:17:53.857897 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.857867 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-bound-sa-token" pod "7082f68a-2079-426e-997f-1d3ba3997ba6" (UID: "7082f68a-2079-426e-997f-1d3ba3997ba6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:17:53.947167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.947137 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46q9r\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-kube-api-access-46q9r\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:17:53.947167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.947166 5055 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7082f68a-2079-426e-997f-1d3ba3997ba6-trusted-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:17:53.947421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.947179 5055 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-installation-pull-secrets\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:17:53.947421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.947195 5055 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7082f68a-2079-426e-997f-1d3ba3997ba6-image-registry-private-configuration\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:17:53.947421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.947208 5055 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-bound-sa-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:17:53.947421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.947216 5055 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7082f68a-2079-426e-997f-1d3ba3997ba6-registry-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:17:53.947421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:53.947225 5055 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7082f68a-2079-426e-997f-1d3ba3997ba6-ca-trust-extracted\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:17:54.154796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:54.154754 5055 generic.go:356] "Generic (PLEG): container finished" podID="7082f68a-2079-426e-997f-1d3ba3997ba6" containerID="525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55" exitCode=0 Oct 30 01:17:54.154796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:54.154799 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-655b484568-p27mb" Oct 30 01:17:54.155045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:54.154844 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655b484568-p27mb" event={"ID":"7082f68a-2079-426e-997f-1d3ba3997ba6","Type":"ContainerDied","Data":"525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55"} Oct 30 01:17:54.155045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:54.154908 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-655b484568-p27mb" event={"ID":"7082f68a-2079-426e-997f-1d3ba3997ba6","Type":"ContainerDied","Data":"a879ec3a72661efac01ed9379c5a826dae53540bb3c8e2ee36adfe094cffa0c0"} Oct 30 01:17:54.155045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:54.154938 5055 scope.go:122] "RemoveContainer" containerID="525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55" Oct 30 01:17:54.164796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:54.164779 5055 scope.go:122] "RemoveContainer" containerID="525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55" Oct 30 01:17:54.165060 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:17:54.165035 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55\": container with ID starting with 525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55 not found: ID does not exist" containerID="525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55" Oct 30 01:17:54.165130 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:54.165075 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55"} err="failed to get container status \"525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55\": rpc error: code = NotFound desc = could not find container \"525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55\": container with ID starting with 525897a1442dbaa9e461119c61a73ad01ba2e0e64d81d93e4a2925fdea12dc55 not found: ID does not exist" Oct 30 01:17:54.209609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:54.209577 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-655b484568-p27mb"] Oct 30 01:17:54.222489 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:54.222459 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-655b484568-p27mb"] Oct 30 01:17:54.817627 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:17:54.817599 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7082f68a-2079-426e-997f-1d3ba3997ba6" path="/var/lib/kubelet/pods/7082f68a-2079-426e-997f-1d3ba3997ba6/volumes" Oct 30 01:18:00.831570 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:00.831538 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-gmfkz-7g8gt"] Oct 30 01:18:00.832051 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:00.831714 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" podUID="5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" containerName="insights-gathering" containerID="cri-o://357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188" gracePeriod=30 Oct 30 01:18:01.142159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.142129 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:18:01.186154 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.186119 5055 generic.go:356] "Generic (PLEG): container finished" podID="5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" containerID="357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188" exitCode=2 Oct 30 01:18:01.186367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.186169 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" Oct 30 01:18:01.186367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.186206 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" event={"ID":"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591","Type":"ContainerDied","Data":"357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188"} Oct 30 01:18:01.186367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.186260 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-gmfkz-7g8gt" event={"ID":"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591","Type":"ContainerDied","Data":"6b3ff3a129302e0236657cddceb47cb1094f1e633f34692359e065d77cd2a394"} Oct 30 01:18:01.186367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.186286 5055 scope.go:122] "RemoveContainer" containerID="357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188" Oct 30 01:18:01.205513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.205492 5055 scope.go:122] "RemoveContainer" containerID="357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188" Oct 30 01:18:01.205839 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:18:01.205816 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188\": container with ID starting with 357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188 not found: ID does not exist" containerID="357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188" Oct 30 01:18:01.205945 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.205852 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188"} err="failed to get container status \"357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188\": rpc error: code = NotFound desc = could not find container \"357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188\": container with ID starting with 357376de25c9d936e384801d5d1141e20868f6dcb317f6d59dc07eff79539188 not found: ID does not exist" Oct 30 01:18:01.311756 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.311724 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-trusted-ca-bundle\") pod \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " Oct 30 01:18:01.311920 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.311787 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-kube-api-access-7vgmg\" (UniqueName: \"kubernetes.io/projected/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-kube-api-access-7vgmg\") pod \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " Oct 30 01:18:01.311920 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.311814 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-archives-path\") pod \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " Oct 30 01:18:01.311920 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.311847 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-service-ca-bundle\") pod \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\" (UID: \"5fc585a4-2d42-40f1-bd8d-c17d9dc7f591\") " Oct 30 01:18:01.312191 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.312165 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-trusted-ca-bundle" pod "5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" (UID: "5fc585a4-2d42-40f1-bd8d-c17d9dc7f591"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:01.312191 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.312175 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-archives-path" pod "5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" (UID: "5fc585a4-2d42-40f1-bd8d-c17d9dc7f591"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:01.312257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.312224 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-service-ca-bundle" pod "5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" (UID: "5fc585a4-2d42-40f1-bd8d-c17d9dc7f591"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:01.314254 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.314233 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-kube-api-access-7vgmg" pod "5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" (UID: "5fc585a4-2d42-40f1-bd8d-c17d9dc7f591"). InnerVolumeSpecName "kube-api-access-7vgmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:01.412743 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.412667 5055 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-trusted-ca-bundle\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:01.412743 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.412699 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vgmg\" (UniqueName: \"kubernetes.io/projected/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-kube-api-access-7vgmg\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:01.412743 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.412709 5055 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-archives-path\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:01.412743 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.412718 5055 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591-service-ca-bundle\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:01.530054 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.530019 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-gmfkz-7g8gt"] Oct 30 01:18:01.542595 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:01.542560 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-gmfkz-7g8gt"] Oct 30 01:18:02.817313 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:02.817283 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" path="/var/lib/kubelet/pods/5fc585a4-2d42-40f1-bd8d-c17d9dc7f591/volumes" Oct 30 01:18:16.175096 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.175053 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" podUID="806fd3b6-00de-4f42-8c35-e638d3835ec7" containerName="registry" containerID="cri-o://2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb" gracePeriod=30 Oct 30 01:18:16.468250 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.468217 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:18:16.535447 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.535417 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-trusted-ca\") pod \"806fd3b6-00de-4f42-8c35-e638d3835ec7\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " Oct 30 01:18:16.535609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.535460 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-bound-sa-token\") pod \"806fd3b6-00de-4f42-8c35-e638d3835ec7\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " Oct 30 01:18:16.535609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.535486 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-certificates\") pod \"806fd3b6-00de-4f42-8c35-e638d3835ec7\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " Oct 30 01:18:16.535609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.535514 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") pod \"806fd3b6-00de-4f42-8c35-e638d3835ec7\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " Oct 30 01:18:16.535722 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.535693 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-kube-api-access-2tqtw\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-kube-api-access-2tqtw\") pod \"806fd3b6-00de-4f42-8c35-e638d3835ec7\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " Oct 30 01:18:16.535773 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.535732 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806fd3b6-00de-4f42-8c35-e638d3835ec7-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/806fd3b6-00de-4f42-8c35-e638d3835ec7-ca-trust-extracted\") pod \"806fd3b6-00de-4f42-8c35-e638d3835ec7\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " Oct 30 01:18:16.535773 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.535762 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-installation-pull-secrets\") pod \"806fd3b6-00de-4f42-8c35-e638d3835ec7\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " Oct 30 01:18:16.535890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.535843 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-image-registry-private-configuration\") pod \"806fd3b6-00de-4f42-8c35-e638d3835ec7\" (UID: \"806fd3b6-00de-4f42-8c35-e638d3835ec7\") " Oct 30 01:18:16.536160 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.536116 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-trusted-ca" pod "806fd3b6-00de-4f42-8c35-e638d3835ec7" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:16.536496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.536415 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-certificates" pod "806fd3b6-00de-4f42-8c35-e638d3835ec7" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:16.539133 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.539010 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-installation-pull-secrets" pod "806fd3b6-00de-4f42-8c35-e638d3835ec7" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:16.539133 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.539065 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls" pod "806fd3b6-00de-4f42-8c35-e638d3835ec7" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:16.539133 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.539065 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-kube-api-access-2tqtw" pod "806fd3b6-00de-4f42-8c35-e638d3835ec7" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7"). InnerVolumeSpecName "kube-api-access-2tqtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:16.539423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.539214 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-bound-sa-token" pod "806fd3b6-00de-4f42-8c35-e638d3835ec7" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:16.539505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.539481 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-image-registry-private-configuration" pod "806fd3b6-00de-4f42-8c35-e638d3835ec7" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:16.544893 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.544868 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806fd3b6-00de-4f42-8c35-e638d3835ec7-ca-trust-extracted" pod "806fd3b6-00de-4f42-8c35-e638d3835ec7" (UID: "806fd3b6-00de-4f42-8c35-e638d3835ec7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:16.637079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.637040 5055 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-trusted-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:16.637079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.637076 5055 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-bound-sa-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:16.637300 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.637090 5055 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-certificates\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:16.637300 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.637106 5055 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-registry-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:16.637300 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.637118 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tqtw\" (UniqueName: \"kubernetes.io/projected/806fd3b6-00de-4f42-8c35-e638d3835ec7-kube-api-access-2tqtw\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:16.637300 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.637128 5055 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/806fd3b6-00de-4f42-8c35-e638d3835ec7-ca-trust-extracted\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:16.637300 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.637142 5055 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-installation-pull-secrets\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:16.637300 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:16.637155 5055 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/806fd3b6-00de-4f42-8c35-e638d3835ec7-image-registry-private-configuration\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:17.251879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:17.251847 5055 generic.go:356] "Generic (PLEG): container finished" podID="806fd3b6-00de-4f42-8c35-e638d3835ec7" containerID="2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb" exitCode=0 Oct 30 01:18:17.252346 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:17.251907 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" Oct 30 01:18:17.252346 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:17.251934 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" event={"ID":"806fd3b6-00de-4f42-8c35-e638d3835ec7","Type":"ContainerDied","Data":"2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb"} Oct 30 01:18:17.252346 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:17.251992 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-54db9d5677-ptwtb" event={"ID":"806fd3b6-00de-4f42-8c35-e638d3835ec7","Type":"ContainerDied","Data":"25be08911bf0ee43bd80e546c4a59e397e8fe230dfcbc2d7647418f28dec9cff"} Oct 30 01:18:17.252346 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:17.252018 5055 scope.go:122] "RemoveContainer" containerID="2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb" Oct 30 01:18:17.262523 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:17.262492 5055 scope.go:122] "RemoveContainer" containerID="2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb" Oct 30 01:18:17.263133 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:18:17.263113 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb\": container with ID starting with 2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb not found: ID does not exist" containerID="2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb" Oct 30 01:18:17.263236 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:17.263141 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb"} err="failed to get container status \"2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb\": rpc error: code = NotFound desc = could not find container \"2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb\": container with ID starting with 2d47932b529a67a6f96225c266a86c1562357d6f3a8a08ec6c3b71c1b1d2d3cb not found: ID does not exist" Oct 30 01:18:17.315986 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:17.315957 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-54db9d5677-ptwtb"] Oct 30 01:18:17.328416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:17.328386 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-54db9d5677-ptwtb"] Oct 30 01:18:18.818306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:18.818278 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806fd3b6-00de-4f42-8c35-e638d3835ec7" path="/var/lib/kubelet/pods/806fd3b6-00de-4f42-8c35-e638d3835ec7/volumes" Oct 30 01:18:19.217109 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:18:19.217072 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:18:49.217507 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:18:49.217466 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:18:55.411062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:55.411029 5055 generic.go:356] "Generic (PLEG): container finished" podID="2592437b-af46-4f92-8db7-0580d36f7f80" containerID="43c2ce6c6b7f59adc83e444dc98f7ca50a8b231d02014a69f1b9a9af7e7f242f" exitCode=0 Oct 30 01:18:55.411506 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:55.411102 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rwmdr-6hm72" event={"ID":"2592437b-af46-4f92-8db7-0580d36f7f80","Type":"ContainerDied","Data":"43c2ce6c6b7f59adc83e444dc98f7ca50a8b231d02014a69f1b9a9af7e7f242f"} Oct 30 01:18:56.562444 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.562420 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:18:56.565433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.565415 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-trusted-ca-bundle\") pod \"2592437b-af46-4f92-8db7-0580d36f7f80\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " Oct 30 01:18:56.565512 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.565447 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2592437b-af46-4f92-8db7-0580d36f7f80-kube-api-access-swzmk\" (UniqueName: \"kubernetes.io/projected/2592437b-af46-4f92-8db7-0580d36f7f80-kube-api-access-swzmk\") pod \"2592437b-af46-4f92-8db7-0580d36f7f80\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " Oct 30 01:18:56.565512 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.565495 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2592437b-af46-4f92-8db7-0580d36f7f80-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2592437b-af46-4f92-8db7-0580d36f7f80-archives-path\") pod \"2592437b-af46-4f92-8db7-0580d36f7f80\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " Oct 30 01:18:56.565617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.565546 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-service-ca-bundle\") pod \"2592437b-af46-4f92-8db7-0580d36f7f80\" (UID: \"2592437b-af46-4f92-8db7-0580d36f7f80\") " Oct 30 01:18:56.565815 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.565784 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-trusted-ca-bundle" pod "2592437b-af46-4f92-8db7-0580d36f7f80" (UID: "2592437b-af46-4f92-8db7-0580d36f7f80"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:56.565815 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.565801 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2592437b-af46-4f92-8db7-0580d36f7f80-archives-path" pod "2592437b-af46-4f92-8db7-0580d36f7f80" (UID: "2592437b-af46-4f92-8db7-0580d36f7f80"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:56.565978 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.565957 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-service-ca-bundle" pod "2592437b-af46-4f92-8db7-0580d36f7f80" (UID: "2592437b-af46-4f92-8db7-0580d36f7f80"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:56.567702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.567682 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2592437b-af46-4f92-8db7-0580d36f7f80-kube-api-access-swzmk" pod "2592437b-af46-4f92-8db7-0580d36f7f80" (UID: "2592437b-af46-4f92-8db7-0580d36f7f80"). InnerVolumeSpecName "kube-api-access-swzmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:56.666135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.666045 5055 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-service-ca-bundle\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:56.666135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.666084 5055 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2592437b-af46-4f92-8db7-0580d36f7f80-trusted-ca-bundle\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:56.666135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.666095 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swzmk\" (UniqueName: \"kubernetes.io/projected/2592437b-af46-4f92-8db7-0580d36f7f80-kube-api-access-swzmk\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:56.666135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:56.666105 5055 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/2592437b-af46-4f92-8db7-0580d36f7f80-archives-path\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:18:57.420210 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:57.420170 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rwmdr-6hm72" event={"ID":"2592437b-af46-4f92-8db7-0580d36f7f80","Type":"ContainerDied","Data":"4806a3aeca46022a6e9dc006e668223c21991080f1229eff6325b825e4a82d7e"} Oct 30 01:18:57.420210 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:57.420210 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rwmdr-6hm72" Oct 30 01:18:57.420416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:18:57.420223 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4806a3aeca46022a6e9dc006e668223c21991080f1229eff6325b825e4a82d7e" Oct 30 01:19:19.217574 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:19:19.217535 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:19:49.216717 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:19:49.216627 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:20:19.216945 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:20:19.216908 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:20:49.217663 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:20:49.217627 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:21:19.218118 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:21:19.218077 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:21:27.653937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.653898 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt"] Oct 30 01:21:27.654501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654480 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7082f68a-2079-426e-997f-1d3ba3997ba6" containerName="registry" Oct 30 01:21:27.654602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654504 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7082f68a-2079-426e-997f-1d3ba3997ba6" containerName="registry" Oct 30 01:21:27.654602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654526 5055 cpu_manager.go:414] "containerMap: removing container" podUID="5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" containerName="insights-gathering" Oct 30 01:21:27.654602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654534 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" containerName="insights-gathering" Oct 30 01:21:27.654602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654567 5055 cpu_manager.go:414] "containerMap: removing container" podUID="806fd3b6-00de-4f42-8c35-e638d3835ec7" containerName="registry" Oct 30 01:21:27.654602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654575 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806fd3b6-00de-4f42-8c35-e638d3835ec7" containerName="registry" Oct 30 01:21:27.654602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654600 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2592437b-af46-4f92-8db7-0580d36f7f80" containerName="insights-gathering" Oct 30 01:21:27.654879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654609 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2592437b-af46-4f92-8db7-0580d36f7f80" containerName="insights-gathering" Oct 30 01:21:27.654879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654719 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7082f68a-2079-426e-997f-1d3ba3997ba6" containerName="registry" Oct 30 01:21:27.654879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654734 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fc585a4-2d42-40f1-bd8d-c17d9dc7f591" containerName="insights-gathering" Oct 30 01:21:27.654879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654751 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="806fd3b6-00de-4f42-8c35-e638d3835ec7" containerName="registry" Oct 30 01:21:27.654879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.654762 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2592437b-af46-4f92-8db7-0580d36f7f80" containerName="insights-gathering" Oct 30 01:21:27.657361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.657315 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.676962 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.676942 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 01:21:27.683624 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.683604 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt"] Oct 30 01:21:27.691777 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.691755 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s9nfd\"" Oct 30 01:21:27.763585 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.763559 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24028870-9648-421c-9e8d-a196c897c985-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.763694 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.763597 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d47gr\" (UniqueName: \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-kube-api-access-d47gr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.763694 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.763676 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.763781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.763734 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.763781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.763754 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.763781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.763770 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.763889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.763794 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.763889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.763836 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24028870-9648-421c-9e8d-a196c897c985-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.763950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.763930 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.787810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.787785 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq"] Oct 30 01:21:27.790263 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.790245 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.818318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.818292 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq"] Oct 30 01:21:27.818910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.818886 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-f8zv4\"" Oct 30 01:21:27.864261 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864234 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.864394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864268 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24028870-9648-421c-9e8d-a196c897c985-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.864394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864285 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d47gr\" (UniqueName: \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-kube-api-access-d47gr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.864502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864393 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.864502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864425 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.864502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864453 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.864659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864534 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.864659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864576 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.864659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864652 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.864822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864674 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.864822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864695 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.864822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864715 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.864822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864747 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.864822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864810 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.865076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864869 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24028870-9648-421c-9e8d-a196c897c985-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.865076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864890 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.865076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864970 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.865076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864980 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b636b75-528c-4c41-8d2a-082205cbfdbb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.865076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.864976 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24028870-9648-421c-9e8d-a196c897c985-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.865076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.865055 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.865076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.865078 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.865399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.865094 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.865399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.865102 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qljq\" (UniqueName: \"kubernetes.io/projected/4b636b75-528c-4c41-8d2a-082205cbfdbb-kube-api-access-5qljq\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.867165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.867137 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.867405 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.867385 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24028870-9648-421c-9e8d-a196c897c985-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.883775 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.883746 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.883912 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.883895 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d47gr\" (UniqueName: \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-kube-api-access-d47gr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8qtbt\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.965919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.965893 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966041 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.965924 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966041 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.965946 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.966151 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966238 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.966189 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966279 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.966237 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b636b75-528c-4c41-8d2a-082205cbfdbb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.966289 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.966299 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.966316 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.966364 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.966521 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qljq\" (UniqueName: \"kubernetes.io/projected/4b636b75-528c-4c41-8d2a-082205cbfdbb-kube-api-access-5qljq\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966677 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.966650 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.966677 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.966670 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.967136 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.967114 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b636b75-528c-4c41-8d2a-082205cbfdbb-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.968308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.968287 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.968658 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.968637 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.971455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.971438 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:27.986777 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.986754 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b636b75-528c-4c41-8d2a-082205cbfdbb-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:27.986880 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:27.986861 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qljq\" (UniqueName: \"kubernetes.io/projected/4b636b75-528c-4c41-8d2a-082205cbfdbb-kube-api-access-5qljq\") pod \"openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq\" (UID: \"4b636b75-528c-4c41-8d2a-082205cbfdbb\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:28.103524 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:28.103483 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:28.283375 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:28.283349 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt"] Oct 30 01:21:28.284017 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:21:28.283991 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24028870_9648_421c_9e8d_a196c897c985.slice/crio-93a337c780073192df6316cf635ef59eba8d49f18f230d81aedea4b3cb6c5493 WatchSource:0}: Error finding container 93a337c780073192df6316cf635ef59eba8d49f18f230d81aedea4b3cb6c5493: Status 404 returned error can't find the container with id 93a337c780073192df6316cf635ef59eba8d49f18f230d81aedea4b3cb6c5493 Oct 30 01:21:28.401565 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:28.401541 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq"] Oct 30 01:21:28.403296 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:21:28.403272 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b636b75_528c_4c41_8d2a_082205cbfdbb.slice/crio-05286046497ee9fe860e6fcd982a961f6be466435dd23b87e900fa3b86141ec0 WatchSource:0}: Error finding container 05286046497ee9fe860e6fcd982a961f6be466435dd23b87e900fa3b86141ec0: Status 404 returned error can't find the container with id 05286046497ee9fe860e6fcd982a961f6be466435dd23b87e900fa3b86141ec0 Oct 30 01:21:29.030965 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:29.030923 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" event={"ID":"4b636b75-528c-4c41-8d2a-082205cbfdbb","Type":"ContainerStarted","Data":"05286046497ee9fe860e6fcd982a961f6be466435dd23b87e900fa3b86141ec0"} Oct 30 01:21:29.048985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:29.048951 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" event={"ID":"24028870-9648-421c-9e8d-a196c897c985","Type":"ContainerStarted","Data":"93a337c780073192df6316cf635ef59eba8d49f18f230d81aedea4b3cb6c5493"} Oct 30 01:21:31.295670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:31.295639 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:21:31.296097 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:31.295745 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:21:31.296097 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:31.295794 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:21:31.742907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:31.742869 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:21:31.743032 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:31.742959 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:21:31.743032 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:31.743003 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:21:32.065736 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:32.065652 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" event={"ID":"4b636b75-528c-4c41-8d2a-082205cbfdbb","Type":"ContainerStarted","Data":"9db7789718a09f7bc65a063b0916ba23a8dfef7f411b4d0ac0eeb96992c1a797"} Oct 30 01:21:32.067138 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:32.067111 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" event={"ID":"24028870-9648-421c-9e8d-a196c897c985","Type":"ContainerStarted","Data":"24852896299b26e419fbfd2f8e09c470a77ae3892d9efb6ae0afd88a703936f5"} Oct 30 01:21:32.104057 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:32.104037 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:32.108585 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:32.108564 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:32.121670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:32.121628 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" podStartSLOduration=2.232221625 podStartE2EDuration="5.121616707s" totalImagesPullingTime="2.889395082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:32.120519392 +0000 UTC m=+259.441476434" watchObservedRunningTime="2026-10-30 01:21:32.121616707 +0000 UTC m=+259.442573746" Oct 30 01:21:32.204869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:32.204827 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" podStartSLOduration=1.748498962 podStartE2EDuration="5.204815814s" totalImagesPullingTime="3.456316852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:32.204133041 +0000 UTC m=+259.525090081" watchObservedRunningTime="2026-10-30 01:21:32.204815814 +0000 UTC m=+259.525772854" Oct 30 01:21:32.971780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:32.971747 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:32.976285 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:32.976264 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:33.072041 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:33.071998 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:33.072187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:33.072060 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:33.072783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:33.072765 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq" Oct 30 01:21:33.072889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:21:33.072816 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:21:49.216933 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:21:49.216898 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:22:11.768079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.768048 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g"] Oct 30 01:22:11.774456 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.774439 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.796935 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.796906 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g"] Oct 30 01:22:11.827248 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.827219 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c19d799c-5a58-4728-a568-9f0fb10afe86-manifests\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.827386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.827257 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c19d799c-5a58-4728-a568-9f0fb10afe86-tmp\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.827386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.827290 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b7gp\" (UniqueName: \"kubernetes.io/projected/c19d799c-5a58-4728-a568-9f0fb10afe86-kube-api-access-4b7gp\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.827494 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.827434 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c19d799c-5a58-4728-a568-9f0fb10afe86-metrics-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.827554 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.827514 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c19d799c-5a58-4728-a568-9f0fb10afe86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.827608 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.827572 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c19d799c-5a58-4728-a568-9f0fb10afe86-webhook-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.847477 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.847450 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 01:22:11.847571 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.847505 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 01:22:11.847623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.847610 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 01:22:11.847676 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.847659 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 01:22:11.847709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.847664 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-pp4hc\"" Oct 30 01:22:11.847709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.847685 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 01:22:11.928486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.928459 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c19d799c-5a58-4728-a568-9f0fb10afe86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.928622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.928492 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c19d799c-5a58-4728-a568-9f0fb10afe86-webhook-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.928622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.928513 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c19d799c-5a58-4728-a568-9f0fb10afe86-manifests\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.928622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.928566 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c19d799c-5a58-4728-a568-9f0fb10afe86-tmp\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.928780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.928627 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b7gp\" (UniqueName: \"kubernetes.io/projected/c19d799c-5a58-4728-a568-9f0fb10afe86-kube-api-access-4b7gp\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.928780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.928675 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c19d799c-5a58-4728-a568-9f0fb10afe86-metrics-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.928927 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.928904 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c19d799c-5a58-4728-a568-9f0fb10afe86-manifests\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.931157 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.931137 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c19d799c-5a58-4728-a568-9f0fb10afe86-tmp\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.931290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.931275 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c19d799c-5a58-4728-a568-9f0fb10afe86-webhook-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.931359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.931347 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c19d799c-5a58-4728-a568-9f0fb10afe86-metrics-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.931560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.931541 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c19d799c-5a58-4728-a568-9f0fb10afe86-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:11.949529 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:11.949510 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b7gp\" (UniqueName: \"kubernetes.io/projected/c19d799c-5a58-4728-a568-9f0fb10afe86-kube-api-access-4b7gp\") pod \"opendatahub-operator-controller-manager-685895cf4f-8hn4g\" (UID: \"c19d799c-5a58-4728-a568-9f0fb10afe86\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:12.088364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:12.088270 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:12.443849 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:22:12.443817 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19d799c_5a58_4728_a568_9f0fb10afe86.slice/crio-f32efafac336adf978e19569410573cfe63b246c549b5357fde61269f7b8f79b WatchSource:0}: Error finding container f32efafac336adf978e19569410573cfe63b246c549b5357fde61269f7b8f79b: Status 404 returned error can't find the container with id f32efafac336adf978e19569410573cfe63b246c549b5357fde61269f7b8f79b Oct 30 01:22:12.444386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:12.444362 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g"] Oct 30 01:22:12.838196 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:12.838172 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:22:12.839948 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:12.839929 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:22:13.232578 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:13.232541 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" event={"ID":"c19d799c-5a58-4728-a568-9f0fb10afe86","Type":"ContainerStarted","Data":"f32efafac336adf978e19569410573cfe63b246c549b5357fde61269f7b8f79b"} Oct 30 01:22:16.247885 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:16.247852 5055 generic.go:356] "Generic (PLEG): container finished" podID="c19d799c-5a58-4728-a568-9f0fb10afe86" containerID="9f217c50a8d4a38b7986b16634a03ceaf31dd30a85f605404b61724ff7495e8c" exitCode=0 Oct 30 01:22:16.248264 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:16.247891 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" event={"ID":"c19d799c-5a58-4728-a568-9f0fb10afe86","Type":"ContainerDied","Data":"9f217c50a8d4a38b7986b16634a03ceaf31dd30a85f605404b61724ff7495e8c"} Oct 30 01:22:16.248457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:16.248441 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:22:17.253833 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:17.253800 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" event={"ID":"c19d799c-5a58-4728-a568-9f0fb10afe86","Type":"ContainerStarted","Data":"161e23dce34a371d61bc8394249b6613b6baea83f91e95b42d35540a903abc9d"} Oct 30 01:22:17.254230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:17.253914 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:17.339555 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:17.339507 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" podStartSLOduration=2.686524103 podStartE2EDuration="6.339494334s" totalImagesPullingTime="3.191618819s" totalInitContainerRuntime="461.351412ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:17.338410409 +0000 UTC m=+304.659367449" watchObservedRunningTime="2026-10-30 01:22:17.339494334 +0000 UTC m=+304.660451371" Oct 30 01:22:19.217854 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:22:19.217811 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:22:28.260533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:28.260457 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-8hn4g" Oct 30 01:22:29.001883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.001850 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g"] Oct 30 01:22:29.005049 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.005034 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g" Oct 30 01:22:29.020918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.020899 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-tv86r\"" Oct 30 01:22:29.021048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.020951 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 01:22:29.021048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.020982 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 01:22:29.021048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.021030 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 01:22:29.030215 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.030194 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g"] Oct 30 01:22:29.177231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.177199 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bglz\" (UniqueName: \"kubernetes.io/projected/7d2c4e13-3895-4202-aef2-b0e726d79e3b-kube-api-access-8bglz\") pod \"dns-operator-controller-manager-8557cbdb84-hdh5g\" (UID: \"7d2c4e13-3895-4202-aef2-b0e726d79e3b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g" Oct 30 01:22:29.277836 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.277771 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bglz\" (UniqueName: \"kubernetes.io/projected/7d2c4e13-3895-4202-aef2-b0e726d79e3b-kube-api-access-8bglz\") pod \"dns-operator-controller-manager-8557cbdb84-hdh5g\" (UID: \"7d2c4e13-3895-4202-aef2-b0e726d79e3b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g" Oct 30 01:22:29.301040 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.301011 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bglz\" (UniqueName: \"kubernetes.io/projected/7d2c4e13-3895-4202-aef2-b0e726d79e3b-kube-api-access-8bglz\") pod \"dns-operator-controller-manager-8557cbdb84-hdh5g\" (UID: \"7d2c4e13-3895-4202-aef2-b0e726d79e3b\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g" Oct 30 01:22:29.316942 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.316923 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g" Oct 30 01:22:29.623542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:29.623514 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g"] Oct 30 01:22:29.624356 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:22:29.624311 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d2c4e13_3895_4202_aef2_b0e726d79e3b.slice/crio-6a824c74be6c71a304d3ae646bf023be95aa938a27d315c3ca01a5aebb84e69a WatchSource:0}: Error finding container 6a824c74be6c71a304d3ae646bf023be95aa938a27d315c3ca01a5aebb84e69a: Status 404 returned error can't find the container with id 6a824c74be6c71a304d3ae646bf023be95aa938a27d315c3ca01a5aebb84e69a Oct 30 01:22:30.309081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:30.309038 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g" event={"ID":"7d2c4e13-3895-4202-aef2-b0e726d79e3b","Type":"ContainerStarted","Data":"6a824c74be6c71a304d3ae646bf023be95aa938a27d315c3ca01a5aebb84e69a"} Oct 30 01:22:32.319793 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:32.319755 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g" event={"ID":"7d2c4e13-3895-4202-aef2-b0e726d79e3b","Type":"ContainerStarted","Data":"5d748791ec5d38d36d63e96d6092338305783bf79fa6acdc6cf6414f6cffe687"} Oct 30 01:22:32.320167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:32.319814 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g" Oct 30 01:22:32.363258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:32.363204 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g" podStartSLOduration=2.063054918 podStartE2EDuration="4.363189734s" totalImagesPullingTime="2.300134816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:32.362576534 +0000 UTC m=+319.683533574" watchObservedRunningTime="2026-10-30 01:22:32.363189734 +0000 UTC m=+319.684146775" Oct 30 01:22:36.256711 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:36.256675 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt"] Oct 30 01:22:36.260265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:36.260243 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt" Oct 30 01:22:36.280174 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:36.280152 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gmqf4\"" Oct 30 01:22:36.293469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:36.293427 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt"] Oct 30 01:22:36.436459 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:36.436426 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qv9w\" (UniqueName: \"kubernetes.io/projected/ee296e37-1ffe-4f80-825b-086fa417666d-kube-api-access-2qv9w\") pod \"authorino-operator-7b7bb4bc77-nz6qt\" (UID: \"ee296e37-1ffe-4f80-825b-086fa417666d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt" Oct 30 01:22:36.537436 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:36.537351 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qv9w\" (UniqueName: \"kubernetes.io/projected/ee296e37-1ffe-4f80-825b-086fa417666d-kube-api-access-2qv9w\") pod \"authorino-operator-7b7bb4bc77-nz6qt\" (UID: \"ee296e37-1ffe-4f80-825b-086fa417666d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt" Oct 30 01:22:36.557862 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:36.557822 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qv9w\" (UniqueName: \"kubernetes.io/projected/ee296e37-1ffe-4f80-825b-086fa417666d-kube-api-access-2qv9w\") pod \"authorino-operator-7b7bb4bc77-nz6qt\" (UID: \"ee296e37-1ffe-4f80-825b-086fa417666d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt" Oct 30 01:22:36.572506 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:36.572473 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt" Oct 30 01:22:36.870096 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:36.870071 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt"] Oct 30 01:22:36.870694 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:22:36.870657 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee296e37_1ffe_4f80_825b_086fa417666d.slice/crio-7ad941a5ccfc2c1dc6f35ff60e5ea46da583eeee08881384ead3fc7aef0b03c5 WatchSource:0}: Error finding container 7ad941a5ccfc2c1dc6f35ff60e5ea46da583eeee08881384ead3fc7aef0b03c5: Status 404 returned error can't find the container with id 7ad941a5ccfc2c1dc6f35ff60e5ea46da583eeee08881384ead3fc7aef0b03c5 Oct 30 01:22:37.342687 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:37.342659 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt" event={"ID":"ee296e37-1ffe-4f80-825b-086fa417666d","Type":"ContainerStarted","Data":"7ad941a5ccfc2c1dc6f35ff60e5ea46da583eeee08881384ead3fc7aef0b03c5"} Oct 30 01:22:39.354216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:39.354179 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt" event={"ID":"ee296e37-1ffe-4f80-825b-086fa417666d","Type":"ContainerStarted","Data":"551d63b04b6df8a517e8b0e2f7b2c38bb4834fe216f1fcd7a15e3efbe8dce492"} Oct 30 01:22:39.354658 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:39.354284 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt" Oct 30 01:22:39.394837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:39.394794 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt" podStartSLOduration=1.6956878629999999 podStartE2EDuration="3.394782616s" totalImagesPullingTime="1.699094753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:39.394527405 +0000 UTC m=+326.715484447" watchObservedRunningTime="2026-10-30 01:22:39.394782616 +0000 UTC m=+326.715739656" Oct 30 01:22:42.622639 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.622602 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5"] Oct 30 01:22:42.626230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.626213 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" Oct 30 01:22:42.644311 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.644280 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5"] Oct 30 01:22:42.655294 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.655271 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4z4jk\"" Oct 30 01:22:42.790187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.790150 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bf33bf9-2177-45f4-9ebd-493c07fb49db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-59kh5\" (UID: \"0bf33bf9-2177-45f4-9ebd-493c07fb49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" Oct 30 01:22:42.790360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.790195 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p45kk\" (UniqueName: \"kubernetes.io/projected/0bf33bf9-2177-45f4-9ebd-493c07fb49db-kube-api-access-p45kk\") pod \"kuadrant-operator-controller-manager-678d87766f-59kh5\" (UID: \"0bf33bf9-2177-45f4-9ebd-493c07fb49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" Oct 30 01:22:42.891235 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.891155 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bf33bf9-2177-45f4-9ebd-493c07fb49db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-59kh5\" (UID: \"0bf33bf9-2177-45f4-9ebd-493c07fb49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" Oct 30 01:22:42.891235 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.891196 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p45kk\" (UniqueName: \"kubernetes.io/projected/0bf33bf9-2177-45f4-9ebd-493c07fb49db-kube-api-access-p45kk\") pod \"kuadrant-operator-controller-manager-678d87766f-59kh5\" (UID: \"0bf33bf9-2177-45f4-9ebd-493c07fb49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" Oct 30 01:22:42.891562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.891542 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0bf33bf9-2177-45f4-9ebd-493c07fb49db-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-59kh5\" (UID: \"0bf33bf9-2177-45f4-9ebd-493c07fb49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" Oct 30 01:22:42.913499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.913474 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p45kk\" (UniqueName: \"kubernetes.io/projected/0bf33bf9-2177-45f4-9ebd-493c07fb49db-kube-api-access-p45kk\") pod \"kuadrant-operator-controller-manager-678d87766f-59kh5\" (UID: \"0bf33bf9-2177-45f4-9ebd-493c07fb49db\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" Oct 30 01:22:42.938853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:42.938833 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" Oct 30 01:22:43.259569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:43.259537 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5"] Oct 30 01:22:43.260192 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:22:43.260170 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf33bf9_2177_45f4_9ebd_493c07fb49db.slice/crio-724278842c42b453966ac54338f9a073e3b84d4ed4cb0a4d48333debf16f7bf3 WatchSource:0}: Error finding container 724278842c42b453966ac54338f9a073e3b84d4ed4cb0a4d48333debf16f7bf3: Status 404 returned error can't find the container with id 724278842c42b453966ac54338f9a073e3b84d4ed4cb0a4d48333debf16f7bf3 Oct 30 01:22:43.326656 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:43.326625 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-hdh5g" Oct 30 01:22:43.373604 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:43.373580 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" event={"ID":"0bf33bf9-2177-45f4-9ebd-493c07fb49db","Type":"ContainerStarted","Data":"724278842c42b453966ac54338f9a073e3b84d4ed4cb0a4d48333debf16f7bf3"} Oct 30 01:22:47.464880 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.464849 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg"] Oct 30 01:22:47.468708 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.468686 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.485925 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.485904 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-fk2rp\"" Oct 30 01:22:47.500858 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.500787 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg"] Oct 30 01:22:47.528446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.528420 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk5b7\" (UniqueName: \"kubernetes.io/projected/551f197c-a32d-4dcb-b4f7-d8106d500e9b-kube-api-access-gk5b7\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.528600 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.528462 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.528600 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.528483 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.528600 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.528500 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.528600 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.528586 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.528787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.528619 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.528787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.528640 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.528787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.528659 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.528787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.528742 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.629475 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.629430 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.629475 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.629475 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.629721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.629497 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.629721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.629540 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk5b7\" (UniqueName: \"kubernetes.io/projected/551f197c-a32d-4dcb-b4f7-d8106d500e9b-kube-api-access-gk5b7\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.629721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.629574 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.629721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.629605 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.629721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.629627 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.629721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.629660 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.629721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.629700 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.630119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.629901 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.630119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.630034 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.630119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.630055 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.630290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.630150 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.630373 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.630292 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.632125 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.632096 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.632317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.632297 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.652786 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.652760 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551f197c-a32d-4dcb-b4f7-d8106d500e9b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.653089 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.653063 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk5b7\" (UniqueName: \"kubernetes.io/projected/551f197c-a32d-4dcb-b4f7-d8106d500e9b-kube-api-access-gk5b7\") pod \"data-science-gateway-data-science-gateway-class-6455b4886fvvsbg\" (UID: \"551f197c-a32d-4dcb-b4f7-d8106d500e9b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:47.782708 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:47.782617 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:48.123048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:48.123022 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg"] Oct 30 01:22:48.125472 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:48.125429 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:22:48.125591 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:48.125518 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:22:48.125591 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:48.125558 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:22:48.398461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:48.398383 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" event={"ID":"551f197c-a32d-4dcb-b4f7-d8106d500e9b","Type":"ContainerStarted","Data":"337fbad44b39c4fb1ee61b8005074e2b61a37b6c0e77c387335aa65d9fe2c7ad"} Oct 30 01:22:48.398461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:48.398433 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" event={"ID":"551f197c-a32d-4dcb-b4f7-d8106d500e9b","Type":"ContainerStarted","Data":"c7eb5eec3ac4d0dc8f40d6cbfea433a7549233cbf88974fd2582db2bb6ff8105"} Oct 30 01:22:48.443989 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:48.443941 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" podStartSLOduration=1.4439252630000001 podStartE2EDuration="1.443925263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:48.443082789 +0000 UTC m=+335.764039820" watchObservedRunningTime="2026-10-30 01:22:48.443925263 +0000 UTC m=+335.764882304" Oct 30 01:22:48.782763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:48.782729 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:49.218407 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:22:49.218370 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:22:49.788277 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:49.788247 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:50.362298 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:50.362269 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-nz6qt" Oct 30 01:22:50.408427 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:50.408311 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:50.409312 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:50.409284 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6455b4886fvvsbg" Oct 30 01:22:52.418274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:52.418232 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" event={"ID":"0bf33bf9-2177-45f4-9ebd-493c07fb49db","Type":"ContainerStarted","Data":"59098f60eae06096a4e3b96357e9bd3fbffc26d36c157eb96e559d6eae9f0af5"} Oct 30 01:22:52.418786 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:52.418451 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" Oct 30 01:22:52.486764 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:22:52.486711 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" podStartSLOduration=2.378798293 podStartE2EDuration="10.486694748s" totalImagesPullingTime="8.107896455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:52.472874537 +0000 UTC m=+339.793831577" watchObservedRunningTime="2026-10-30 01:22:52.486694748 +0000 UTC m=+339.807651790" Oct 30 01:23:03.424587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:03.424551 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-59kh5" Oct 30 01:23:04.857822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:04.857794 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sxgcm"] Oct 30 01:23:04.883527 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:04.883498 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:04.884341 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:04.884303 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sxgcm"] Oct 30 01:23:04.900867 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:04.900593 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 01:23:04.912657 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:04.912632 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7tx6b\"" Oct 30 01:23:04.979556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:04.979506 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1c8fc01-3340-48d0-b44c-70deae82230c-config-file\") pod \"limitador-limitador-74dfcfb894-sxgcm\" (UID: \"d1c8fc01-3340-48d0-b44c-70deae82230c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:04.979556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:04.979566 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf4fc\" (UniqueName: \"kubernetes.io/projected/d1c8fc01-3340-48d0-b44c-70deae82230c-kube-api-access-xf4fc\") pod \"limitador-limitador-74dfcfb894-sxgcm\" (UID: \"d1c8fc01-3340-48d0-b44c-70deae82230c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:04.989134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:04.989108 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sxgcm"] Oct 30 01:23:05.081094 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:05.081063 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1c8fc01-3340-48d0-b44c-70deae82230c-config-file\") pod \"limitador-limitador-74dfcfb894-sxgcm\" (UID: \"d1c8fc01-3340-48d0-b44c-70deae82230c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:05.081280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:05.081127 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf4fc\" (UniqueName: \"kubernetes.io/projected/d1c8fc01-3340-48d0-b44c-70deae82230c-kube-api-access-xf4fc\") pod \"limitador-limitador-74dfcfb894-sxgcm\" (UID: \"d1c8fc01-3340-48d0-b44c-70deae82230c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:05.081784 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:05.081761 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1c8fc01-3340-48d0-b44c-70deae82230c-config-file\") pod \"limitador-limitador-74dfcfb894-sxgcm\" (UID: \"d1c8fc01-3340-48d0-b44c-70deae82230c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:05.108385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:05.108303 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf4fc\" (UniqueName: \"kubernetes.io/projected/d1c8fc01-3340-48d0-b44c-70deae82230c-kube-api-access-xf4fc\") pod \"limitador-limitador-74dfcfb894-sxgcm\" (UID: \"d1c8fc01-3340-48d0-b44c-70deae82230c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:05.212026 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:05.211987 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:05.517368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:05.517308 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sxgcm"] Oct 30 01:23:05.517918 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:23:05.517889 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c8fc01_3340_48d0_b44c_70deae82230c.slice/crio-426db6111ee2ff3ede3847274703b2c35e30ff96a58db7806f19f0b894dda658 WatchSource:0}: Error finding container 426db6111ee2ff3ede3847274703b2c35e30ff96a58db7806f19f0b894dda658: Status 404 returned error can't find the container with id 426db6111ee2ff3ede3847274703b2c35e30ff96a58db7806f19f0b894dda658 Oct 30 01:23:06.221182 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:06.221151 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5t445"] Oct 30 01:23:06.224121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:06.224102 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5t445" Oct 30 01:23:06.243418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:06.243393 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5t445"] Oct 30 01:23:06.254006 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:06.253986 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-q9rn8\"" Oct 30 01:23:06.291781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:06.291751 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvgnd\" (UniqueName: \"kubernetes.io/projected/915c78ce-11f9-400e-ba0e-c610ae46f500-kube-api-access-mvgnd\") pod \"authorino-56c6c9ddd5-5t445\" (UID: \"915c78ce-11f9-400e-ba0e-c610ae46f500\") " pod="kuadrant-system/authorino-56c6c9ddd5-5t445" Oct 30 01:23:06.393048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:06.393021 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvgnd\" (UniqueName: \"kubernetes.io/projected/915c78ce-11f9-400e-ba0e-c610ae46f500-kube-api-access-mvgnd\") pod \"authorino-56c6c9ddd5-5t445\" (UID: \"915c78ce-11f9-400e-ba0e-c610ae46f500\") " pod="kuadrant-system/authorino-56c6c9ddd5-5t445" Oct 30 01:23:06.412905 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:06.412876 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvgnd\" (UniqueName: \"kubernetes.io/projected/915c78ce-11f9-400e-ba0e-c610ae46f500-kube-api-access-mvgnd\") pod \"authorino-56c6c9ddd5-5t445\" (UID: \"915c78ce-11f9-400e-ba0e-c610ae46f500\") " pod="kuadrant-system/authorino-56c6c9ddd5-5t445" Oct 30 01:23:06.481880 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:06.481824 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" event={"ID":"d1c8fc01-3340-48d0-b44c-70deae82230c","Type":"ContainerStarted","Data":"426db6111ee2ff3ede3847274703b2c35e30ff96a58db7806f19f0b894dda658"} Oct 30 01:23:06.553757 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:06.553734 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5t445" Oct 30 01:23:06.847742 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:23:06.847703 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod915c78ce_11f9_400e_ba0e_c610ae46f500.slice/crio-6bbeeec40cbbe503fd621da28ce9eba7855603d76c6ea164792e2cd37b3d5106 WatchSource:0}: Error finding container 6bbeeec40cbbe503fd621da28ce9eba7855603d76c6ea164792e2cd37b3d5106: Status 404 returned error can't find the container with id 6bbeeec40cbbe503fd621da28ce9eba7855603d76c6ea164792e2cd37b3d5106 Oct 30 01:23:06.847951 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:06.847925 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5t445"] Oct 30 01:23:07.488644 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:07.488610 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" event={"ID":"d1c8fc01-3340-48d0-b44c-70deae82230c","Type":"ContainerStarted","Data":"082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074"} Oct 30 01:23:07.489132 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:07.488695 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:07.489880 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:07.489855 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5t445" event={"ID":"915c78ce-11f9-400e-ba0e-c610ae46f500","Type":"ContainerStarted","Data":"6bbeeec40cbbe503fd621da28ce9eba7855603d76c6ea164792e2cd37b3d5106"} Oct 30 01:23:07.530283 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:07.530234 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" podStartSLOduration=1.940592908 podStartE2EDuration="3.53021398s" totalImagesPullingTime="1.589621072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:07.529580718 +0000 UTC m=+354.850537758" watchObservedRunningTime="2026-10-30 01:23:07.53021398 +0000 UTC m=+354.851171017" Oct 30 01:23:10.508833 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:10.508792 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5t445" event={"ID":"915c78ce-11f9-400e-ba0e-c610ae46f500","Type":"ContainerStarted","Data":"d0936d652d5493df78d07ec701282c54bba92e9f499f2e98b727a48f83f30570"} Oct 30 01:23:10.548273 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:10.548218 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-5t445" podStartSLOduration=1.7114130090000002 podStartE2EDuration="4.548201888s" totalImagesPullingTime="2.836788879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:10.547225525 +0000 UTC m=+357.868182568" watchObservedRunningTime="2026-10-30 01:23:10.548201888 +0000 UTC m=+357.869158928" Oct 30 01:23:18.497470 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:18.497436 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:19.217855 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:23:19.217819 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:23:19.740804 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:19.740766 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sxgcm"] Oct 30 01:23:19.741214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:19.740994 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" podUID="d1c8fc01-3340-48d0-b44c-70deae82230c" containerName="limitador" containerID="cri-o://082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074" gracePeriod=30 Oct 30 01:23:20.290473 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.290446 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:20.424815 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.424728 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c8fc01-3340-48d0-b44c-70deae82230c-kube-api-access-xf4fc\" (UniqueName: \"kubernetes.io/projected/d1c8fc01-3340-48d0-b44c-70deae82230c-kube-api-access-xf4fc\") pod \"d1c8fc01-3340-48d0-b44c-70deae82230c\" (UID: \"d1c8fc01-3340-48d0-b44c-70deae82230c\") " Oct 30 01:23:20.424815 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.424785 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c8fc01-3340-48d0-b44c-70deae82230c-config-file\" (UniqueName: \"kubernetes.io/configmap/d1c8fc01-3340-48d0-b44c-70deae82230c-config-file\") pod \"d1c8fc01-3340-48d0-b44c-70deae82230c\" (UID: \"d1c8fc01-3340-48d0-b44c-70deae82230c\") " Oct 30 01:23:20.425190 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.425155 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c8fc01-3340-48d0-b44c-70deae82230c-config-file" pod "d1c8fc01-3340-48d0-b44c-70deae82230c" (UID: "d1c8fc01-3340-48d0-b44c-70deae82230c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:20.427113 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.427090 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c8fc01-3340-48d0-b44c-70deae82230c-kube-api-access-xf4fc" pod "d1c8fc01-3340-48d0-b44c-70deae82230c" (UID: "d1c8fc01-3340-48d0-b44c-70deae82230c"). InnerVolumeSpecName "kube-api-access-xf4fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:20.525845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.525810 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf4fc\" (UniqueName: \"kubernetes.io/projected/d1c8fc01-3340-48d0-b44c-70deae82230c-kube-api-access-xf4fc\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:23:20.525845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.525844 5055 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d1c8fc01-3340-48d0-b44c-70deae82230c-config-file\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:23:20.557996 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.557959 5055 generic.go:356] "Generic (PLEG): container finished" podID="d1c8fc01-3340-48d0-b44c-70deae82230c" containerID="082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074" exitCode=0 Oct 30 01:23:20.558168 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.558014 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" Oct 30 01:23:20.558168 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.558039 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" event={"ID":"d1c8fc01-3340-48d0-b44c-70deae82230c","Type":"ContainerDied","Data":"082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074"} Oct 30 01:23:20.558168 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.558088 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-sxgcm" event={"ID":"d1c8fc01-3340-48d0-b44c-70deae82230c","Type":"ContainerDied","Data":"426db6111ee2ff3ede3847274703b2c35e30ff96a58db7806f19f0b894dda658"} Oct 30 01:23:20.558168 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.558112 5055 scope.go:122] "RemoveContainer" containerID="082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074" Oct 30 01:23:20.567048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.567026 5055 scope.go:122] "RemoveContainer" containerID="082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074" Oct 30 01:23:20.567319 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:23:20.567302 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074\": container with ID starting with 082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074 not found: ID does not exist" containerID="082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074" Oct 30 01:23:20.567418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.567399 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074"} err="failed to get container status \"082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074\": rpc error: code = NotFound desc = could not find container \"082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074\": container with ID starting with 082c76901110cb2189385c18fb6e92689b921e34cb6ddbfbfff6c7edfc1a0074 not found: ID does not exist" Oct 30 01:23:20.610142 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.610102 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sxgcm"] Oct 30 01:23:20.614238 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.614211 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-sxgcm"] Oct 30 01:23:20.818554 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:20.818524 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c8fc01-3340-48d0-b44c-70deae82230c" path="/var/lib/kubelet/pods/d1c8fc01-3340-48d0-b44c-70deae82230c/volumes" Oct 30 01:23:25.121599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.121563 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt"] Oct 30 01:23:25.122003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.121958 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d1c8fc01-3340-48d0-b44c-70deae82230c" containerName="limitador" Oct 30 01:23:25.122003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.121970 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c8fc01-3340-48d0-b44c-70deae82230c" containerName="limitador" Oct 30 01:23:25.122110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.122037 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c8fc01-3340-48d0-b44c-70deae82230c" containerName="limitador" Oct 30 01:23:25.124113 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.124090 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" Oct 30 01:23:25.144682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.144653 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zpp5p\"" Oct 30 01:23:25.149561 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.149510 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 01:23:25.157875 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.157846 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:23:25.158039 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.157887 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:23:25.159520 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.159498 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt"] Oct 30 01:23:25.267890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.267851 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfklz\" (UniqueName: \"kubernetes.io/projected/a32eb4fc-8b4a-4afa-b95a-29497eee0df4-kube-api-access-jfklz\") pod \"postgres-7cdbf6dc44-m77kt\" (UID: \"a32eb4fc-8b4a-4afa-b95a-29497eee0df4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" Oct 30 01:23:25.268074 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.267937 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a32eb4fc-8b4a-4afa-b95a-29497eee0df4-data\") pod \"postgres-7cdbf6dc44-m77kt\" (UID: \"a32eb4fc-8b4a-4afa-b95a-29497eee0df4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" Oct 30 01:23:25.369088 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.369053 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a32eb4fc-8b4a-4afa-b95a-29497eee0df4-data\") pod \"postgres-7cdbf6dc44-m77kt\" (UID: \"a32eb4fc-8b4a-4afa-b95a-29497eee0df4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" Oct 30 01:23:25.369260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.369138 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfklz\" (UniqueName: \"kubernetes.io/projected/a32eb4fc-8b4a-4afa-b95a-29497eee0df4-kube-api-access-jfklz\") pod \"postgres-7cdbf6dc44-m77kt\" (UID: \"a32eb4fc-8b4a-4afa-b95a-29497eee0df4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" Oct 30 01:23:25.369532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.369511 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a32eb4fc-8b4a-4afa-b95a-29497eee0df4-data\") pod \"postgres-7cdbf6dc44-m77kt\" (UID: \"a32eb4fc-8b4a-4afa-b95a-29497eee0df4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" Oct 30 01:23:25.389835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.389765 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfklz\" (UniqueName: \"kubernetes.io/projected/a32eb4fc-8b4a-4afa-b95a-29497eee0df4-kube-api-access-jfklz\") pod \"postgres-7cdbf6dc44-m77kt\" (UID: \"a32eb4fc-8b4a-4afa-b95a-29497eee0df4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" Oct 30 01:23:25.436827 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.436794 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" Oct 30 01:23:25.740988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:25.740961 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt"] Oct 30 01:23:25.742371 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:23:25.742342 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda32eb4fc_8b4a_4afa_b95a_29497eee0df4.slice/crio-a159c3d99ce9c7c1d9ccaf319b61361d75dc1d6d7c4512b36f238a134f0b2b35 WatchSource:0}: Error finding container a159c3d99ce9c7c1d9ccaf319b61361d75dc1d6d7c4512b36f238a134f0b2b35: Status 404 returned error can't find the container with id a159c3d99ce9c7c1d9ccaf319b61361d75dc1d6d7c4512b36f238a134f0b2b35 Oct 30 01:23:26.589027 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:26.588991 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" event={"ID":"a32eb4fc-8b4a-4afa-b95a-29497eee0df4","Type":"ContainerStarted","Data":"a159c3d99ce9c7c1d9ccaf319b61361d75dc1d6d7c4512b36f238a134f0b2b35"} Oct 30 01:23:31.616535 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:31.616504 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" event={"ID":"a32eb4fc-8b4a-4afa-b95a-29497eee0df4","Type":"ContainerStarted","Data":"8cc389aa1a1f2c67a014d1bb02f034f32a0adfbe077328efbffbcec1a9bddd34"} Oct 30 01:23:31.617000 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:31.616566 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" Oct 30 01:23:31.655522 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:31.655479 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" podStartSLOduration=1.685020945 podStartE2EDuration="6.655465218s" totalImagesPullingTime="4.970444273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:31.654811216 +0000 UTC m=+378.975768256" watchObservedRunningTime="2026-10-30 01:23:31.655465218 +0000 UTC m=+378.976422258" Oct 30 01:23:37.650526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:37.650498 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-m77kt" Oct 30 01:23:44.705073 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.705043 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86c788b6bc-tgrsp"] Oct 30 01:23:44.709503 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.709480 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:44.726544 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.726519 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 01:23:44.726651 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.726559 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 01:23:44.731446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.731428 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-metrics-certs\") pod \"maas-controller-86c788b6bc-tgrsp\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:44.731523 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.731493 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjzzm\" (UniqueName: \"kubernetes.io/projected/374fb861-abd7-4ad7-bc97-1b662134316b-kube-api-access-gjzzm\") pod \"maas-controller-86c788b6bc-tgrsp\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:44.731608 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.731592 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-webhook-cert\") pod \"maas-controller-86c788b6bc-tgrsp\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:44.739337 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.739302 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 01:23:44.739475 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.739360 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 01:23:44.739623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.739605 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 01:23:44.739723 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.739699 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-nlskm\"" Oct 30 01:23:44.741457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.741438 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86c788b6bc-tgrsp"] Oct 30 01:23:44.832339 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.832297 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-metrics-certs\") pod \"maas-controller-86c788b6bc-tgrsp\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:44.832499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.832370 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjzzm\" (UniqueName: \"kubernetes.io/projected/374fb861-abd7-4ad7-bc97-1b662134316b-kube-api-access-gjzzm\") pod \"maas-controller-86c788b6bc-tgrsp\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:44.832499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.832420 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-webhook-cert\") pod \"maas-controller-86c788b6bc-tgrsp\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:44.834935 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.834913 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-metrics-certs\") pod \"maas-controller-86c788b6bc-tgrsp\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:44.835052 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.834994 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-webhook-cert\") pod \"maas-controller-86c788b6bc-tgrsp\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:44.857887 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:44.857858 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjzzm\" (UniqueName: \"kubernetes.io/projected/374fb861-abd7-4ad7-bc97-1b662134316b-kube-api-access-gjzzm\") pod \"maas-controller-86c788b6bc-tgrsp\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:45.022005 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.021912 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:45.290367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.290284 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78cc6d847c-4nbvj"] Oct 30 01:23:45.294266 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.294251 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.333033 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.333008 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78cc6d847c-4nbvj"] Oct 30 01:23:45.336827 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.336801 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m885\" (UniqueName: \"kubernetes.io/projected/b505b494-3911-42aa-80f7-71cfece2b6da-kube-api-access-8m885\") pod \"maas-controller-78cc6d847c-4nbvj\" (UID: \"b505b494-3911-42aa-80f7-71cfece2b6da\") " pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.336954 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.336915 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b505b494-3911-42aa-80f7-71cfece2b6da-webhook-cert\") pod \"maas-controller-78cc6d847c-4nbvj\" (UID: \"b505b494-3911-42aa-80f7-71cfece2b6da\") " pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.337022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.336957 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b505b494-3911-42aa-80f7-71cfece2b6da-metrics-certs\") pod \"maas-controller-78cc6d847c-4nbvj\" (UID: \"b505b494-3911-42aa-80f7-71cfece2b6da\") " pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.372629 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.372607 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86c788b6bc-tgrsp"] Oct 30 01:23:45.373179 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:23:45.373156 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374fb861_abd7_4ad7_bc97_1b662134316b.slice/crio-d043529bd68549a071011279c9254d2402ed47c9cb3dc389bcdeaa950990a58f WatchSource:0}: Error finding container d043529bd68549a071011279c9254d2402ed47c9cb3dc389bcdeaa950990a58f: Status 404 returned error can't find the container with id d043529bd68549a071011279c9254d2402ed47c9cb3dc389bcdeaa950990a58f Oct 30 01:23:45.438402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.438376 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m885\" (UniqueName: \"kubernetes.io/projected/b505b494-3911-42aa-80f7-71cfece2b6da-kube-api-access-8m885\") pod \"maas-controller-78cc6d847c-4nbvj\" (UID: \"b505b494-3911-42aa-80f7-71cfece2b6da\") " pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.438511 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.438432 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b505b494-3911-42aa-80f7-71cfece2b6da-webhook-cert\") pod \"maas-controller-78cc6d847c-4nbvj\" (UID: \"b505b494-3911-42aa-80f7-71cfece2b6da\") " pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.438511 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.438468 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b505b494-3911-42aa-80f7-71cfece2b6da-metrics-certs\") pod \"maas-controller-78cc6d847c-4nbvj\" (UID: \"b505b494-3911-42aa-80f7-71cfece2b6da\") " pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.440911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.440887 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b505b494-3911-42aa-80f7-71cfece2b6da-webhook-cert\") pod \"maas-controller-78cc6d847c-4nbvj\" (UID: \"b505b494-3911-42aa-80f7-71cfece2b6da\") " pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.440996 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.440919 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b505b494-3911-42aa-80f7-71cfece2b6da-metrics-certs\") pod \"maas-controller-78cc6d847c-4nbvj\" (UID: \"b505b494-3911-42aa-80f7-71cfece2b6da\") " pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.458588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.458570 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m885\" (UniqueName: \"kubernetes.io/projected/b505b494-3911-42aa-80f7-71cfece2b6da-kube-api-access-8m885\") pod \"maas-controller-78cc6d847c-4nbvj\" (UID: \"b505b494-3911-42aa-80f7-71cfece2b6da\") " pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.607076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.607018 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:45.683598 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.683564 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" event={"ID":"374fb861-abd7-4ad7-bc97-1b662134316b","Type":"ContainerStarted","Data":"d043529bd68549a071011279c9254d2402ed47c9cb3dc389bcdeaa950990a58f"} Oct 30 01:23:45.928527 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:45.928497 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78cc6d847c-4nbvj"] Oct 30 01:23:45.929098 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:23:45.929065 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb505b494_3911_42aa_80f7_71cfece2b6da.slice/crio-25554952e97514c9ec0c545572e93403324611f707212e8f273b1115faba07b4 WatchSource:0}: Error finding container 25554952e97514c9ec0c545572e93403324611f707212e8f273b1115faba07b4: Status 404 returned error can't find the container with id 25554952e97514c9ec0c545572e93403324611f707212e8f273b1115faba07b4 Oct 30 01:23:46.699713 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:46.699671 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78cc6d847c-4nbvj" event={"ID":"b505b494-3911-42aa-80f7-71cfece2b6da","Type":"ContainerStarted","Data":"25554952e97514c9ec0c545572e93403324611f707212e8f273b1115faba07b4"} Oct 30 01:23:48.718447 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:48.718403 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" event={"ID":"374fb861-abd7-4ad7-bc97-1b662134316b","Type":"ContainerStarted","Data":"006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281"} Oct 30 01:23:48.719228 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:48.718485 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:48.720887 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:48.720851 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78cc6d847c-4nbvj" event={"ID":"b505b494-3911-42aa-80f7-71cfece2b6da","Type":"ContainerStarted","Data":"01076d47ea388c349b7fe70fc08f6713eff5f14ecd188bbea2cfcd748ff31356"} Oct 30 01:23:48.721047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:48.721017 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:48.771249 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:48.771158 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" podStartSLOduration=1.947351099 podStartE2EDuration="4.771145024s" totalImagesPullingTime="2.823793925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:48.769656669 +0000 UTC m=+396.090613711" watchObservedRunningTime="2026-10-30 01:23:48.771145024 +0000 UTC m=+396.092102055" Oct 30 01:23:48.815012 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:48.814962 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-78cc6d847c-4nbvj" podStartSLOduration=1.260659633 podStartE2EDuration="3.814944283s" totalImagesPullingTime="2.55428465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:48.81386931 +0000 UTC m=+396.134826351" watchObservedRunningTime="2026-10-30 01:23:48.814944283 +0000 UTC m=+396.135901324" Oct 30 01:23:49.219009 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:23:49.218960 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:23:52.034960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.034928 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jhxpw"] Oct 30 01:23:52.038528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.038512 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" Oct 30 01:23:52.057631 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.057606 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 01:23:52.057631 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.057633 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zjf5f\"" Oct 30 01:23:52.061921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.061896 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jhxpw"] Oct 30 01:23:52.071208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.071180 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz"] Oct 30 01:23:52.076036 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.075964 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:23:52.097603 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.097569 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwm2m\" (UniqueName: \"kubernetes.io/projected/d0d27620-b33d-46c5-bbb3-e33ea7abb823-kube-api-access-cwm2m\") pod \"llmisvc-controller-manager-869d5dcc5d-2q7nz\" (UID: \"d0d27620-b33d-46c5-bbb3-e33ea7abb823\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:23:52.097774 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.097724 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0d27620-b33d-46c5-bbb3-e33ea7abb823-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2q7nz\" (UID: \"d0d27620-b33d-46c5-bbb3-e33ea7abb823\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:23:52.097774 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.097762 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58571a89-2cf1-44fa-8ef4-a052d02e1a82-cert\") pod \"kserve-controller-manager-776cc66d47-jhxpw\" (UID: \"58571a89-2cf1-44fa-8ef4-a052d02e1a82\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" Oct 30 01:23:52.097910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.097891 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hn94\" (UniqueName: \"kubernetes.io/projected/58571a89-2cf1-44fa-8ef4-a052d02e1a82-kube-api-access-9hn94\") pod \"kserve-controller-manager-776cc66d47-jhxpw\" (UID: \"58571a89-2cf1-44fa-8ef4-a052d02e1a82\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" Oct 30 01:23:52.100625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.100598 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz"] Oct 30 01:23:52.119652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.119610 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cbpcp"] Oct 30 01:23:52.123441 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.123413 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" Oct 30 01:23:52.151643 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.151612 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cbpcp"] Oct 30 01:23:52.154123 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.154093 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 01:23:52.154299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.154284 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 01:23:52.154819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.154455 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-nfjsv\"" Oct 30 01:23:52.154819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.154506 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-qmh89\"" Oct 30 01:23:52.159196 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.159153 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-92htv"] Oct 30 01:23:52.163201 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.163166 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.183318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.183286 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-92htv"] Oct 30 01:23:52.199404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.199368 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a93ddb49-90a6-416d-981a-afd68b09076b-cert\") pod \"odh-model-controller-769875db56-92htv\" (UID: \"a93ddb49-90a6-416d-981a-afd68b09076b\") " pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.199592 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.199483 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hn94\" (UniqueName: \"kubernetes.io/projected/58571a89-2cf1-44fa-8ef4-a052d02e1a82-kube-api-access-9hn94\") pod \"kserve-controller-manager-776cc66d47-jhxpw\" (UID: \"58571a89-2cf1-44fa-8ef4-a052d02e1a82\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" Oct 30 01:23:52.199592 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.199524 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dce57b6-a313-4051-a33b-7300d36d13ab-tls-certs\") pod \"model-serving-api-75bd89fdfc-cbpcp\" (UID: \"5dce57b6-a313-4051-a33b-7300d36d13ab\") " pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" Oct 30 01:23:52.199592 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.199560 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwqsp\" (UniqueName: \"kubernetes.io/projected/a93ddb49-90a6-416d-981a-afd68b09076b-kube-api-access-bwqsp\") pod \"odh-model-controller-769875db56-92htv\" (UID: \"a93ddb49-90a6-416d-981a-afd68b09076b\") " pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.199592 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.199583 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwm2m\" (UniqueName: \"kubernetes.io/projected/d0d27620-b33d-46c5-bbb3-e33ea7abb823-kube-api-access-cwm2m\") pod \"llmisvc-controller-manager-869d5dcc5d-2q7nz\" (UID: \"d0d27620-b33d-46c5-bbb3-e33ea7abb823\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:23:52.199754 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.199606 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93ddb49-90a6-416d-981a-afd68b09076b-metrics-certs\") pod \"odh-model-controller-769875db56-92htv\" (UID: \"a93ddb49-90a6-416d-981a-afd68b09076b\") " pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.199830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.199802 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0d27620-b33d-46c5-bbb3-e33ea7abb823-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2q7nz\" (UID: \"d0d27620-b33d-46c5-bbb3-e33ea7abb823\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:23:52.199868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.199852 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwsdl\" (UniqueName: \"kubernetes.io/projected/5dce57b6-a313-4051-a33b-7300d36d13ab-kube-api-access-cwsdl\") pod \"model-serving-api-75bd89fdfc-cbpcp\" (UID: \"5dce57b6-a313-4051-a33b-7300d36d13ab\") " pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" Oct 30 01:23:52.199905 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.199895 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58571a89-2cf1-44fa-8ef4-a052d02e1a82-cert\") pod \"kserve-controller-manager-776cc66d47-jhxpw\" (UID: \"58571a89-2cf1-44fa-8ef4-a052d02e1a82\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" Oct 30 01:23:52.202419 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.202395 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0d27620-b33d-46c5-bbb3-e33ea7abb823-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-2q7nz\" (UID: \"d0d27620-b33d-46c5-bbb3-e33ea7abb823\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:23:52.202569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.202552 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/58571a89-2cf1-44fa-8ef4-a052d02e1a82-cert\") pod \"kserve-controller-manager-776cc66d47-jhxpw\" (UID: \"58571a89-2cf1-44fa-8ef4-a052d02e1a82\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" Oct 30 01:23:52.222232 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.222198 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwm2m\" (UniqueName: \"kubernetes.io/projected/d0d27620-b33d-46c5-bbb3-e33ea7abb823-kube-api-access-cwm2m\") pod \"llmisvc-controller-manager-869d5dcc5d-2q7nz\" (UID: \"d0d27620-b33d-46c5-bbb3-e33ea7abb823\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:23:52.222479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.222459 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hn94\" (UniqueName: \"kubernetes.io/projected/58571a89-2cf1-44fa-8ef4-a052d02e1a82-kube-api-access-9hn94\") pod \"kserve-controller-manager-776cc66d47-jhxpw\" (UID: \"58571a89-2cf1-44fa-8ef4-a052d02e1a82\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" Oct 30 01:23:52.255036 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.255008 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-l4r8d\"" Oct 30 01:23:52.255216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.255095 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 01:23:52.255216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.255154 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 01:23:52.300831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.300725 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a93ddb49-90a6-416d-981a-afd68b09076b-cert\") pod \"odh-model-controller-769875db56-92htv\" (UID: \"a93ddb49-90a6-416d-981a-afd68b09076b\") " pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.300831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.300785 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dce57b6-a313-4051-a33b-7300d36d13ab-tls-certs\") pod \"model-serving-api-75bd89fdfc-cbpcp\" (UID: \"5dce57b6-a313-4051-a33b-7300d36d13ab\") " pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" Oct 30 01:23:52.300831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.300812 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwqsp\" (UniqueName: \"kubernetes.io/projected/a93ddb49-90a6-416d-981a-afd68b09076b-kube-api-access-bwqsp\") pod \"odh-model-controller-769875db56-92htv\" (UID: \"a93ddb49-90a6-416d-981a-afd68b09076b\") " pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.300831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.300830 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93ddb49-90a6-416d-981a-afd68b09076b-metrics-certs\") pod \"odh-model-controller-769875db56-92htv\" (UID: \"a93ddb49-90a6-416d-981a-afd68b09076b\") " pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.301166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.300877 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwsdl\" (UniqueName: \"kubernetes.io/projected/5dce57b6-a313-4051-a33b-7300d36d13ab-kube-api-access-cwsdl\") pod \"model-serving-api-75bd89fdfc-cbpcp\" (UID: \"5dce57b6-a313-4051-a33b-7300d36d13ab\") " pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" Oct 30 01:23:52.303566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.303538 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a93ddb49-90a6-416d-981a-afd68b09076b-metrics-certs\") pod \"odh-model-controller-769875db56-92htv\" (UID: \"a93ddb49-90a6-416d-981a-afd68b09076b\") " pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.303674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.303544 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a93ddb49-90a6-416d-981a-afd68b09076b-cert\") pod \"odh-model-controller-769875db56-92htv\" (UID: \"a93ddb49-90a6-416d-981a-afd68b09076b\") " pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.303674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.303595 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dce57b6-a313-4051-a33b-7300d36d13ab-tls-certs\") pod \"model-serving-api-75bd89fdfc-cbpcp\" (UID: \"5dce57b6-a313-4051-a33b-7300d36d13ab\") " pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" Oct 30 01:23:52.321756 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.321723 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwqsp\" (UniqueName: \"kubernetes.io/projected/a93ddb49-90a6-416d-981a-afd68b09076b-kube-api-access-bwqsp\") pod \"odh-model-controller-769875db56-92htv\" (UID: \"a93ddb49-90a6-416d-981a-afd68b09076b\") " pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.321914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.321762 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwsdl\" (UniqueName: \"kubernetes.io/projected/5dce57b6-a313-4051-a33b-7300d36d13ab-kube-api-access-cwsdl\") pod \"model-serving-api-75bd89fdfc-cbpcp\" (UID: \"5dce57b6-a313-4051-a33b-7300d36d13ab\") " pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" Oct 30 01:23:52.352499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.352466 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" Oct 30 01:23:52.390243 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.390200 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:23:52.437349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.437281 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" Oct 30 01:23:52.484955 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.484714 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:52.650780 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:23:52.650752 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58571a89_2cf1_44fa_8ef4_a052d02e1a82.slice/crio-826ed08ea120c5155bdd55ffa0e5578947bea819f989df44d1f19adc9863b126 WatchSource:0}: Error finding container 826ed08ea120c5155bdd55ffa0e5578947bea819f989df44d1f19adc9863b126: Status 404 returned error can't find the container with id 826ed08ea120c5155bdd55ffa0e5578947bea819f989df44d1f19adc9863b126 Oct 30 01:23:52.650904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.650873 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jhxpw"] Oct 30 01:23:52.700890 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:23:52.700858 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd0d27620_b33d_46c5_bbb3_e33ea7abb823.slice/crio-af45dfab6250fe774dac9c828dfb47dfaaf3dd1d41410ef321bf6dc6ff8837a4 WatchSource:0}: Error finding container af45dfab6250fe774dac9c828dfb47dfaaf3dd1d41410ef321bf6dc6ff8837a4: Status 404 returned error can't find the container with id af45dfab6250fe774dac9c828dfb47dfaaf3dd1d41410ef321bf6dc6ff8837a4 Oct 30 01:23:52.701312 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.701290 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz"] Oct 30 01:23:52.743234 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.743196 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" event={"ID":"d0d27620-b33d-46c5-bbb3-e33ea7abb823","Type":"ContainerStarted","Data":"af45dfab6250fe774dac9c828dfb47dfaaf3dd1d41410ef321bf6dc6ff8837a4"} Oct 30 01:23:52.744853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.744725 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" event={"ID":"58571a89-2cf1-44fa-8ef4-a052d02e1a82","Type":"ContainerStarted","Data":"826ed08ea120c5155bdd55ffa0e5578947bea819f989df44d1f19adc9863b126"} Oct 30 01:23:52.747021 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.747001 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-cbpcp"] Oct 30 01:23:52.748047 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:23:52.748028 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dce57b6_a313_4051_a33b_7300d36d13ab.slice/crio-34fbe5f27552f74eafa18ac938205dddb275bc75bfb61c3612aed287854aeabb WatchSource:0}: Error finding container 34fbe5f27552f74eafa18ac938205dddb275bc75bfb61c3612aed287854aeabb: Status 404 returned error can't find the container with id 34fbe5f27552f74eafa18ac938205dddb275bc75bfb61c3612aed287854aeabb Oct 30 01:23:52.795036 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:52.795015 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-92htv"] Oct 30 01:23:52.796339 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:23:52.796297 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93ddb49_90a6_416d_981a_afd68b09076b.slice/crio-abe3677324fe6c2cdae8ba4f5a39a70d738582a7f0bf742925805f668f9b4edb WatchSource:0}: Error finding container abe3677324fe6c2cdae8ba4f5a39a70d738582a7f0bf742925805f668f9b4edb: Status 404 returned error can't find the container with id abe3677324fe6c2cdae8ba4f5a39a70d738582a7f0bf742925805f668f9b4edb Oct 30 01:23:53.239807 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.239775 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s"] Oct 30 01:23:53.243865 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.243828 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.267064 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.267020 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 30 01:23:53.270513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.270429 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s"] Oct 30 01:23:53.287649 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.287621 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 30 01:23:53.287832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.287652 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 30 01:23:53.287832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.287753 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-jckcm\"" Oct 30 01:23:53.310699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.310398 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/916568e2-0b52-47e0-a7bd-3635c0ceca97-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.310699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.310499 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/916568e2-0b52-47e0-a7bd-3635c0ceca97-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.310699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.310534 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/916568e2-0b52-47e0-a7bd-3635c0ceca97-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.310699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.310566 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnlrv\" (UniqueName: \"kubernetes.io/projected/916568e2-0b52-47e0-a7bd-3635c0ceca97-kube-api-access-gnlrv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.413853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.411529 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/916568e2-0b52-47e0-a7bd-3635c0ceca97-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.413853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.411644 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/916568e2-0b52-47e0-a7bd-3635c0ceca97-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.413853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.411678 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/916568e2-0b52-47e0-a7bd-3635c0ceca97-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.413853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.411710 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlrv\" (UniqueName: \"kubernetes.io/projected/916568e2-0b52-47e0-a7bd-3635c0ceca97-kube-api-access-gnlrv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.413853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.412630 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/916568e2-0b52-47e0-a7bd-3635c0ceca97-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.413853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.413381 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/916568e2-0b52-47e0-a7bd-3635c0ceca97-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.413853 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:23:53.413507 5055 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 30 01:23:53.413853 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:23:53.413572 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/916568e2-0b52-47e0-a7bd-3635c0ceca97-tls-cert podName:916568e2-0b52-47e0-a7bd-3635c0ceca97 nodeName:}" failed. No retries permitted until 2026-10-30 01:23:53.913550388 +0000 UTC m=+401.234507425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/916568e2-0b52-47e0-a7bd-3635c0ceca97-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-tf79s" (UID: "916568e2-0b52-47e0-a7bd-3635c0ceca97") : secret "usage-logs-tenancy-proxy-tls" not found Oct 30 01:23:53.442255 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.442188 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnlrv\" (UniqueName: \"kubernetes.io/projected/916568e2-0b52-47e0-a7bd-3635c0ceca97-kube-api-access-gnlrv\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.759896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.759856 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" event={"ID":"5dce57b6-a313-4051-a33b-7300d36d13ab","Type":"ContainerStarted","Data":"34fbe5f27552f74eafa18ac938205dddb275bc75bfb61c3612aed287854aeabb"} Oct 30 01:23:53.765066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.765025 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-92htv" event={"ID":"a93ddb49-90a6-416d-981a-afd68b09076b","Type":"ContainerStarted","Data":"abe3677324fe6c2cdae8ba4f5a39a70d738582a7f0bf742925805f668f9b4edb"} Oct 30 01:23:53.918111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.918047 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/916568e2-0b52-47e0-a7bd-3635c0ceca97-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:53.928382 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:53.928351 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/916568e2-0b52-47e0-a7bd-3635c0ceca97-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-tf79s\" (UID: \"916568e2-0b52-47e0-a7bd-3635c0ceca97\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:54.178080 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:54.177707 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:23:54.573163 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:54.573133 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s"] Oct 30 01:23:54.788244 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:54.788200 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" event={"ID":"916568e2-0b52-47e0-a7bd-3635c0ceca97","Type":"ContainerStarted","Data":"79e1307e959830b45c786548362746ec80f273a96f7312a70b7fd0f2317bd577"} Oct 30 01:23:56.824454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:56.824370 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" event={"ID":"5dce57b6-a313-4051-a33b-7300d36d13ab","Type":"ContainerStarted","Data":"b06e56cc53b9db7bb3fdbbcbefcf8fc25828602f8da2a100b4db77b4aaaffdb6"} Oct 30 01:23:56.824454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:56.824441 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" Oct 30 01:23:56.858966 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:56.858867 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" podStartSLOduration=1.120590719 podStartE2EDuration="4.858847147s" totalImagesPullingTime="3.738256428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:56.857156503 +0000 UTC m=+404.178113548" watchObservedRunningTime="2026-10-30 01:23:56.858847147 +0000 UTC m=+404.179804187" Oct 30 01:23:58.832993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:58.832949 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" event={"ID":"58571a89-2cf1-44fa-8ef4-a052d02e1a82","Type":"ContainerStarted","Data":"e590f000d889ec7b71935bcf7117ae215a2c9547de6e538364b8f948f2a00f07"} Oct 30 01:23:58.833473 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:58.833193 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" Oct 30 01:23:58.837356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:58.837303 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-92htv" event={"ID":"a93ddb49-90a6-416d-981a-afd68b09076b","Type":"ContainerStarted","Data":"2f1aaa5d4501b4e43904855307ff540f1f91afbaf0378b6d62129dbbf2a77e97"} Oct 30 01:23:58.837881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:58.837858 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:23:58.876106 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:58.876043 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" podStartSLOduration=0.802910139 podStartE2EDuration="6.876021487s" totalImagesPullingTime="6.073111348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:58.872524875 +0000 UTC m=+406.193481920" watchObservedRunningTime="2026-10-30 01:23:58.876021487 +0000 UTC m=+406.196978528" Oct 30 01:23:58.914609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:58.914544 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-92htv" podStartSLOduration=1.003539084 podStartE2EDuration="6.914521745s" totalImagesPullingTime="5.910982661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:58.912159568 +0000 UTC m=+406.233116611" watchObservedRunningTime="2026-10-30 01:23:58.914521745 +0000 UTC m=+406.235478786" Oct 30 01:23:59.730394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:59.730363 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:23:59.730583 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:59.730442 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78cc6d847c-4nbvj" Oct 30 01:23:59.816568 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:59.816534 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86c788b6bc-tgrsp"] Oct 30 01:23:59.849282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:59.849241 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" podUID="374fb861-abd7-4ad7-bc97-1b662134316b" containerName="manager" containerID="cri-o://006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281" gracePeriod=30 Oct 30 01:23:59.849902 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:59.849829 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" event={"ID":"d0d27620-b33d-46c5-bbb3-e33ea7abb823","Type":"ContainerStarted","Data":"941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9"} Oct 30 01:23:59.850135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:59.850118 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:23:59.938921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:23:59.938858 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" podStartSLOduration=1.919475292 podStartE2EDuration="7.938839087s" totalImagesPullingTime="6.019363795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:59.938496463 +0000 UTC m=+407.259453510" watchObservedRunningTime="2026-10-30 01:23:59.938839087 +0000 UTC m=+407.259796137" Oct 30 01:24:00.757341 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.757301 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:24:00.804556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.804032 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-webhook-cert\" (UniqueName: \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-webhook-cert\") pod \"374fb861-abd7-4ad7-bc97-1b662134316b\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " Oct 30 01:24:00.804556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.804092 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/374fb861-abd7-4ad7-bc97-1b662134316b-kube-api-access-gjzzm\" (UniqueName: \"kubernetes.io/projected/374fb861-abd7-4ad7-bc97-1b662134316b-kube-api-access-gjzzm\") pod \"374fb861-abd7-4ad7-bc97-1b662134316b\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " Oct 30 01:24:00.804891 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.804862 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-metrics-certs\") pod \"374fb861-abd7-4ad7-bc97-1b662134316b\" (UID: \"374fb861-abd7-4ad7-bc97-1b662134316b\") " Oct 30 01:24:00.807678 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.807650 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-webhook-cert" pod "374fb861-abd7-4ad7-bc97-1b662134316b" (UID: "374fb861-abd7-4ad7-bc97-1b662134316b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:00.808428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.808390 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-metrics-certs" pod "374fb861-abd7-4ad7-bc97-1b662134316b" (UID: "374fb861-abd7-4ad7-bc97-1b662134316b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:00.809130 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.809104 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374fb861-abd7-4ad7-bc97-1b662134316b-kube-api-access-gjzzm" pod "374fb861-abd7-4ad7-bc97-1b662134316b" (UID: "374fb861-abd7-4ad7-bc97-1b662134316b"). InnerVolumeSpecName "kube-api-access-gjzzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:00.855529 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.855500 5055 generic.go:356] "Generic (PLEG): container finished" podID="374fb861-abd7-4ad7-bc97-1b662134316b" containerID="006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281" exitCode=0 Oct 30 01:24:00.855939 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.855868 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" event={"ID":"374fb861-abd7-4ad7-bc97-1b662134316b","Type":"ContainerDied","Data":"006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281"} Oct 30 01:24:00.856013 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.855998 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" event={"ID":"374fb861-abd7-4ad7-bc97-1b662134316b","Type":"ContainerDied","Data":"d043529bd68549a071011279c9254d2402ed47c9cb3dc389bcdeaa950990a58f"} Oct 30 01:24:00.856048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.856033 5055 scope.go:122] "RemoveContainer" containerID="006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281" Oct 30 01:24:00.856239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.856210 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86c788b6bc-tgrsp" Oct 30 01:24:00.869460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.869437 5055 scope.go:122] "RemoveContainer" containerID="006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281" Oct 30 01:24:00.869794 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:24:00.869768 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281\": container with ID starting with 006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281 not found: ID does not exist" containerID="006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281" Oct 30 01:24:00.869890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.869808 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281"} err="failed to get container status \"006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281\": rpc error: code = NotFound desc = could not find container \"006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281\": container with ID starting with 006e68e8887e8a0cbc51b73ee6df7bf32f8bf19097b15c0f09ed8b39b0df6281 not found: ID does not exist" Oct 30 01:24:00.906406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.906380 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.906406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.906409 5055 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/374fb861-abd7-4ad7-bc97-1b662134316b-webhook-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.906568 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.906424 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjzzm\" (UniqueName: \"kubernetes.io/projected/374fb861-abd7-4ad7-bc97-1b662134316b-kube-api-access-gjzzm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:24:00.908622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.908594 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86c788b6bc-tgrsp"] Oct 30 01:24:00.914981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:00.914962 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86c788b6bc-tgrsp"] Oct 30 01:24:02.820076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:02.820036 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="374fb861-abd7-4ad7-bc97-1b662134316b" path="/var/lib/kubelet/pods/374fb861-abd7-4ad7-bc97-1b662134316b/volumes" Oct 30 01:24:06.899617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:06.899528 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" event={"ID":"916568e2-0b52-47e0-a7bd-3635c0ceca97","Type":"ContainerStarted","Data":"b681076f84d7aafb2a15f41d8727a3b11ba0cd6111a5e7293304123451739296"} Oct 30 01:24:06.899617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:06.899604 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:24:06.939789 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:06.939731 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" podStartSLOduration=1.906602125 podStartE2EDuration="13.939716485s" totalImagesPullingTime="12.03311436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:06.93868125 +0000 UTC m=+414.259638291" watchObservedRunningTime="2026-10-30 01:24:06.939716485 +0000 UTC m=+414.260673525" Oct 30 01:24:07.830055 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:07.830025 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-cbpcp" Oct 30 01:24:10.859492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:10.859455 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-92htv" Oct 30 01:24:17.910823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:17.910791 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-tf79s" Oct 30 01:24:19.217817 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:24:19.217770 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:24:29.855550 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:29.855522 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-jhxpw" Oct 30 01:24:29.890472 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:29.890424 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxc8x"] Oct 30 01:24:29.890871 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:29.890857 5055 cpu_manager.go:414] "containerMap: removing container" podUID="374fb861-abd7-4ad7-bc97-1b662134316b" containerName="manager" Oct 30 01:24:29.890914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:29.890873 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="374fb861-abd7-4ad7-bc97-1b662134316b" containerName="manager" Oct 30 01:24:29.890979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:29.890970 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="374fb861-abd7-4ad7-bc97-1b662134316b" containerName="manager" Oct 30 01:24:29.957714 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:29.957684 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxc8x"] Oct 30 01:24:29.957874 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:29.957780 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" Oct 30 01:24:29.973152 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:29.973128 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 01:24:30.041883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:30.041855 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxc8x"] Oct 30 01:24:30.042091 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:24:30.042072 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jxtml tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" podUID="3dac3b55-8047-4670-ac02-67c328affe9e" Oct 30 01:24:30.079178 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:30.079147 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxtml\" (UniqueName: \"kubernetes.io/projected/3dac3b55-8047-4670-ac02-67c328affe9e-kube-api-access-jxtml\") pod \"authorino-6889cbd8c4-rxc8x\" (UID: \"3dac3b55-8047-4670-ac02-67c328affe9e\") " pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" Oct 30 01:24:30.079356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:30.079185 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3dac3b55-8047-4670-ac02-67c328affe9e-tls-cert\") pod \"authorino-6889cbd8c4-rxc8x\" (UID: \"3dac3b55-8047-4670-ac02-67c328affe9e\") " pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" Oct 30 01:24:30.180601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:30.180509 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxtml\" (UniqueName: \"kubernetes.io/projected/3dac3b55-8047-4670-ac02-67c328affe9e-kube-api-access-jxtml\") pod \"authorino-6889cbd8c4-rxc8x\" (UID: \"3dac3b55-8047-4670-ac02-67c328affe9e\") " pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" Oct 30 01:24:30.180601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:30.180563 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3dac3b55-8047-4670-ac02-67c328affe9e-tls-cert\") pod \"authorino-6889cbd8c4-rxc8x\" (UID: \"3dac3b55-8047-4670-ac02-67c328affe9e\") " pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" Oct 30 01:24:30.183766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:30.183733 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3dac3b55-8047-4670-ac02-67c328affe9e-tls-cert\") pod \"authorino-6889cbd8c4-rxc8x\" (UID: \"3dac3b55-8047-4670-ac02-67c328affe9e\") " pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" Oct 30 01:24:30.201558 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:30.201527 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxtml\" (UniqueName: \"kubernetes.io/projected/3dac3b55-8047-4670-ac02-67c328affe9e-kube-api-access-jxtml\") pod \"authorino-6889cbd8c4-rxc8x\" (UID: \"3dac3b55-8047-4670-ac02-67c328affe9e\") " pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" Oct 30 01:24:30.859141 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:30.859108 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:24:31.026232 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:31.026195 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" Oct 30 01:24:31.052603 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:31.052570 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" Oct 30 01:24:31.190523 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:31.190448 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dac3b55-8047-4670-ac02-67c328affe9e-kube-api-access-jxtml\" (UniqueName: \"kubernetes.io/projected/3dac3b55-8047-4670-ac02-67c328affe9e-kube-api-access-jxtml\") pod \"3dac3b55-8047-4670-ac02-67c328affe9e\" (UID: \"3dac3b55-8047-4670-ac02-67c328affe9e\") " Oct 30 01:24:31.190666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:31.190623 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dac3b55-8047-4670-ac02-67c328affe9e-tls-cert\" (UniqueName: \"kubernetes.io/secret/3dac3b55-8047-4670-ac02-67c328affe9e-tls-cert\") pod \"3dac3b55-8047-4670-ac02-67c328affe9e\" (UID: \"3dac3b55-8047-4670-ac02-67c328affe9e\") " Oct 30 01:24:31.192877 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:31.192848 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dac3b55-8047-4670-ac02-67c328affe9e-tls-cert" pod "3dac3b55-8047-4670-ac02-67c328affe9e" (UID: "3dac3b55-8047-4670-ac02-67c328affe9e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:31.192877 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:31.192866 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dac3b55-8047-4670-ac02-67c328affe9e-kube-api-access-jxtml" pod "3dac3b55-8047-4670-ac02-67c328affe9e" (UID: "3dac3b55-8047-4670-ac02-67c328affe9e"). InnerVolumeSpecName "kube-api-access-jxtml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:31.291937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:31.291899 5055 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3dac3b55-8047-4670-ac02-67c328affe9e-tls-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.291937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:31.291933 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxtml\" (UniqueName: \"kubernetes.io/projected/3dac3b55-8047-4670-ac02-67c328affe9e-kube-api-access-jxtml\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:24:31.890014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:31.889977 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5t445"] Oct 30 01:24:31.890610 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:31.890199 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-5t445" podUID="915c78ce-11f9-400e-ba0e-c610ae46f500" containerName="authorino" containerID="cri-o://d0936d652d5493df78d07ec701282c54bba92e9f499f2e98b727a48f83f30570" gracePeriod=30 Oct 30 01:24:32.034417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.034383 5055 generic.go:356] "Generic (PLEG): container finished" podID="24028870-9648-421c-9e8d-a196c897c985" containerID="24852896299b26e419fbfd2f8e09c470a77ae3892d9efb6ae0afd88a703936f5" exitCode=0 Oct 30 01:24:32.034587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.034455 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" event={"ID":"24028870-9648-421c-9e8d-a196c897c985","Type":"ContainerDied","Data":"24852896299b26e419fbfd2f8e09c470a77ae3892d9efb6ae0afd88a703936f5"} Oct 30 01:24:32.034869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.034853 5055 scope.go:122] "RemoveContainer" containerID="24852896299b26e419fbfd2f8e09c470a77ae3892d9efb6ae0afd88a703936f5" Oct 30 01:24:32.037985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.036440 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:24:32.037985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.036540 5055 generic.go:356] "Generic (PLEG): container finished" podID="915c78ce-11f9-400e-ba0e-c610ae46f500" containerID="d0936d652d5493df78d07ec701282c54bba92e9f499f2e98b727a48f83f30570" exitCode=0 Oct 30 01:24:32.037985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.036674 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:24:32.037985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.036699 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-rxc8x" Oct 30 01:24:32.037985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.036799 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:24:32.037985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.037097 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5t445" event={"ID":"915c78ce-11f9-400e-ba0e-c610ae46f500","Type":"ContainerDied","Data":"d0936d652d5493df78d07ec701282c54bba92e9f499f2e98b727a48f83f30570"} Oct 30 01:24:32.145963 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.139039 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxc8x"] Oct 30 01:24:32.145963 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.142156 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-rxc8x"] Oct 30 01:24:32.249210 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.249181 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5t445" Oct 30 01:24:32.309285 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.309210 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/915c78ce-11f9-400e-ba0e-c610ae46f500-kube-api-access-mvgnd\" (UniqueName: \"kubernetes.io/projected/915c78ce-11f9-400e-ba0e-c610ae46f500-kube-api-access-mvgnd\") pod \"915c78ce-11f9-400e-ba0e-c610ae46f500\" (UID: \"915c78ce-11f9-400e-ba0e-c610ae46f500\") " Oct 30 01:24:32.311542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.311505 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/915c78ce-11f9-400e-ba0e-c610ae46f500-kube-api-access-mvgnd" pod "915c78ce-11f9-400e-ba0e-c610ae46f500" (UID: "915c78ce-11f9-400e-ba0e-c610ae46f500"). InnerVolumeSpecName "kube-api-access-mvgnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:32.410007 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.409972 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvgnd\" (UniqueName: \"kubernetes.io/projected/915c78ce-11f9-400e-ba0e-c610ae46f500-kube-api-access-mvgnd\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:24:32.819001 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:32.818969 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dac3b55-8047-4670-ac02-67c328affe9e" path="/var/lib/kubelet/pods/3dac3b55-8047-4670-ac02-67c328affe9e/volumes" Oct 30 01:24:33.043067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:33.042939 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-5t445" Oct 30 01:24:33.043067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:33.043029 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-5t445" event={"ID":"915c78ce-11f9-400e-ba0e-c610ae46f500","Type":"ContainerDied","Data":"6bbeeec40cbbe503fd621da28ce9eba7855603d76c6ea164792e2cd37b3d5106"} Oct 30 01:24:33.043799 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:33.043086 5055 scope.go:122] "RemoveContainer" containerID="d0936d652d5493df78d07ec701282c54bba92e9f499f2e98b727a48f83f30570" Oct 30 01:24:33.046147 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:33.046125 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" event={"ID":"24028870-9648-421c-9e8d-a196c897c985","Type":"ContainerStarted","Data":"0233fb7ba8eddd6affbd30046f87eaffb0aa47ba1fcdead31443a7120a9655de"} Oct 30 01:24:33.081401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:33.077972 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:24:33.104794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:33.104766 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5t445"] Oct 30 01:24:33.109514 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:33.109491 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-5t445"] Oct 30 01:24:33.979353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:33.977067 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:24:33.979353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:33.977223 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8qtbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 01:24:33.979353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:33.977281 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 01:24:34.819385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:34.819357 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="915c78ce-11f9-400e-ba0e-c610ae46f500" path="/var/lib/kubelet/pods/915c78ce-11f9-400e-ba0e-c610ae46f500/volumes" Oct 30 01:24:34.982352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:34.980657 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8qtbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 01:24:34.982352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:34.980711 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 01:24:35.973364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:35.973220 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8qtbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 01:24:35.973364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:35.973288 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 01:24:36.983694 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:36.982686 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8qtbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 01:24:36.983694 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:36.982747 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 01:24:37.972160 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:37.972130 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8qtbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 01:24:37.972365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:37.972183 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 01:24:38.975440 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:38.975405 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:24:38.976368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:38.976347 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:24:49.217457 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:24:49.217423 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:24:52.984309 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:52.984278 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp"] Oct 30 01:24:52.984894 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:52.984876 5055 cpu_manager.go:414] "containerMap: removing container" podUID="915c78ce-11f9-400e-ba0e-c610ae46f500" containerName="authorino" Oct 30 01:24:52.984954 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:52.984896 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="915c78ce-11f9-400e-ba0e-c610ae46f500" containerName="authorino" Oct 30 01:24:52.984996 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:52.984987 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="915c78ce-11f9-400e-ba0e-c610ae46f500" containerName="authorino" Oct 30 01:24:53.017296 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:53.017266 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp"] Oct 30 01:24:53.017455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:53.017387 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" Oct 30 01:24:53.082148 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:53.082122 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36ce2879-a790-49fb-85d7-e6cd921a42c1-cert\") pod \"llmisvc-controller-manager-6466c554d8-cvjxp\" (UID: \"36ce2879-a790-49fb-85d7-e6cd921a42c1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" Oct 30 01:24:53.082282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:53.082198 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wr85\" (UniqueName: \"kubernetes.io/projected/36ce2879-a790-49fb-85d7-e6cd921a42c1-kube-api-access-2wr85\") pod \"llmisvc-controller-manager-6466c554d8-cvjxp\" (UID: \"36ce2879-a790-49fb-85d7-e6cd921a42c1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" Oct 30 01:24:53.182781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:53.182752 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wr85\" (UniqueName: \"kubernetes.io/projected/36ce2879-a790-49fb-85d7-e6cd921a42c1-kube-api-access-2wr85\") pod \"llmisvc-controller-manager-6466c554d8-cvjxp\" (UID: \"36ce2879-a790-49fb-85d7-e6cd921a42c1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" Oct 30 01:24:53.182928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:53.182808 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36ce2879-a790-49fb-85d7-e6cd921a42c1-cert\") pod \"llmisvc-controller-manager-6466c554d8-cvjxp\" (UID: \"36ce2879-a790-49fb-85d7-e6cd921a42c1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" Oct 30 01:24:53.185465 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:53.185443 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/36ce2879-a790-49fb-85d7-e6cd921a42c1-cert\") pod \"llmisvc-controller-manager-6466c554d8-cvjxp\" (UID: \"36ce2879-a790-49fb-85d7-e6cd921a42c1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" Oct 30 01:24:53.202315 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:53.202291 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wr85\" (UniqueName: \"kubernetes.io/projected/36ce2879-a790-49fb-85d7-e6cd921a42c1-kube-api-access-2wr85\") pod \"llmisvc-controller-manager-6466c554d8-cvjxp\" (UID: \"36ce2879-a790-49fb-85d7-e6cd921a42c1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" Oct 30 01:24:53.330056 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:53.329977 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" Oct 30 01:24:53.626538 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:24:53.626503 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ce2879_a790_49fb_85d7_e6cd921a42c1.slice/crio-8b82bf94c71c232a1a66c7488cbeeed31de5d45621547ca5f31f56e56273f581 WatchSource:0}: Error finding container 8b82bf94c71c232a1a66c7488cbeeed31de5d45621547ca5f31f56e56273f581: Status 404 returned error can't find the container with id 8b82bf94c71c232a1a66c7488cbeeed31de5d45621547ca5f31f56e56273f581 Oct 30 01:24:53.626792 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:53.626774 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp"] Oct 30 01:24:54.182561 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:54.182526 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" event={"ID":"36ce2879-a790-49fb-85d7-e6cd921a42c1","Type":"ContainerStarted","Data":"5d6e0bd00177ec9e276ed27ccf3ea748299013c093f784baa7cc0f6c0ebd821a"} Oct 30 01:24:54.183015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:54.182609 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" event={"ID":"36ce2879-a790-49fb-85d7-e6cd921a42c1","Type":"ContainerStarted","Data":"8b82bf94c71c232a1a66c7488cbeeed31de5d45621547ca5f31f56e56273f581"} Oct 30 01:24:54.183015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:54.182648 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" Oct 30 01:24:54.221577 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:24:54.221535 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" podStartSLOduration=1.827844926 podStartE2EDuration="2.221523165s" totalImagesPullingTime="393.678239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:54.220877383 +0000 UTC m=+461.541834423" watchObservedRunningTime="2026-10-30 01:24:54.221523165 +0000 UTC m=+461.542480204" Oct 30 01:25:19.216874 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:25:19.216832 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:25:25.190023 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:25.189948 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-cvjxp" Oct 30 01:25:25.254473 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:25.254443 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz"] Oct 30 01:25:25.254727 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:25.254693 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" podUID="d0d27620-b33d-46c5-bbb3-e33ea7abb823" containerName="manager" containerID="cri-o://941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9" gracePeriod=30 Oct 30 01:25:25.505382 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:25.505362 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:25:25.668460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:25.668426 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d27620-b33d-46c5-bbb3-e33ea7abb823-kube-api-access-cwm2m\" (UniqueName: \"kubernetes.io/projected/d0d27620-b33d-46c5-bbb3-e33ea7abb823-kube-api-access-cwm2m\") pod \"d0d27620-b33d-46c5-bbb3-e33ea7abb823\" (UID: \"d0d27620-b33d-46c5-bbb3-e33ea7abb823\") " Oct 30 01:25:25.668625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:25.668480 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d27620-b33d-46c5-bbb3-e33ea7abb823-cert\" (UniqueName: \"kubernetes.io/secret/d0d27620-b33d-46c5-bbb3-e33ea7abb823-cert\") pod \"d0d27620-b33d-46c5-bbb3-e33ea7abb823\" (UID: \"d0d27620-b33d-46c5-bbb3-e33ea7abb823\") " Oct 30 01:25:25.670708 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:25.670679 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d27620-b33d-46c5-bbb3-e33ea7abb823-cert" pod "d0d27620-b33d-46c5-bbb3-e33ea7abb823" (UID: "d0d27620-b33d-46c5-bbb3-e33ea7abb823"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:25.670814 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:25.670725 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d27620-b33d-46c5-bbb3-e33ea7abb823-kube-api-access-cwm2m" pod "d0d27620-b33d-46c5-bbb3-e33ea7abb823" (UID: "d0d27620-b33d-46c5-bbb3-e33ea7abb823"). InnerVolumeSpecName "kube-api-access-cwm2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:25.769807 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:25.769752 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwm2m\" (UniqueName: \"kubernetes.io/projected/d0d27620-b33d-46c5-bbb3-e33ea7abb823-kube-api-access-cwm2m\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:25:25.769807 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:25.769774 5055 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d0d27620-b33d-46c5-bbb3-e33ea7abb823-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:25:26.339052 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:26.339008 5055 generic.go:356] "Generic (PLEG): container finished" podID="d0d27620-b33d-46c5-bbb3-e33ea7abb823" containerID="941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9" exitCode=0 Oct 30 01:25:26.339575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:26.339059 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" Oct 30 01:25:26.339575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:26.339089 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" event={"ID":"d0d27620-b33d-46c5-bbb3-e33ea7abb823","Type":"ContainerDied","Data":"941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9"} Oct 30 01:25:26.339575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:26.339170 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz" event={"ID":"d0d27620-b33d-46c5-bbb3-e33ea7abb823","Type":"ContainerDied","Data":"af45dfab6250fe774dac9c828dfb47dfaaf3dd1d41410ef321bf6dc6ff8837a4"} Oct 30 01:25:26.339575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:26.339206 5055 scope.go:122] "RemoveContainer" containerID="941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9" Oct 30 01:25:26.351067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:26.351045 5055 scope.go:122] "RemoveContainer" containerID="941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9" Oct 30 01:25:26.351402 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:25:26.351377 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9\": container with ID starting with 941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9 not found: ID does not exist" containerID="941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9" Oct 30 01:25:26.351474 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:26.351415 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9"} err="failed to get container status \"941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9\": rpc error: code = NotFound desc = could not find container \"941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9\": container with ID starting with 941b00cdd20f522c3435cf67a09563d3e80249791a01dbf89a3e570f8500a8e9 not found: ID does not exist" Oct 30 01:25:26.389336 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:26.389301 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz"] Oct 30 01:25:26.401574 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:26.401551 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-2q7nz"] Oct 30 01:25:26.818404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:25:26.818373 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d27620-b33d-46c5-bbb3-e33ea7abb823" path="/var/lib/kubelet/pods/d0d27620-b33d-46c5-bbb3-e33ea7abb823/volumes" Oct 30 01:25:49.218036 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:25:49.217999 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:26:19.225470 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:26:19.225404 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:26:49.217549 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:26:49.217512 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:26:58.433480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.433383 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz"] Oct 30 01:26:58.434064 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.434041 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d0d27620-b33d-46c5-bbb3-e33ea7abb823" containerName="manager" Oct 30 01:26:58.434152 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.434067 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d27620-b33d-46c5-bbb3-e33ea7abb823" containerName="manager" Oct 30 01:26:58.434243 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.434230 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d27620-b33d-46c5-bbb3-e33ea7abb823" containerName="manager" Oct 30 01:26:58.437340 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.437306 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.473399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.473320 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz"] Oct 30 01:26:58.573656 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.573617 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dblxx\" (UniqueName: \"kubernetes.io/projected/b367a50a-f9ee-4f39-896b-6baf0615bb8b-kube-api-access-dblxx\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.573819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.573752 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.573819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.573786 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.573819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.573807 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.573956 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.573830 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.573956 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.573942 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.574038 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.574015 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.574071 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.574043 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.574071 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.574061 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.675457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.675423 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.675638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.675471 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.675638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.675502 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.675638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.675538 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.675638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.675566 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.675638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.675594 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dblxx\" (UniqueName: \"kubernetes.io/projected/b367a50a-f9ee-4f39-896b-6baf0615bb8b-kube-api-access-dblxx\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.675917 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.675725 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.675917 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.675774 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.676019 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.675981 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.676076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.676039 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.676132 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.676111 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.676309 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.676284 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.676458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.676440 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.676661 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.676639 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.678027 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.678004 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.678267 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.678250 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.694928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.694872 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dblxx\" (UniqueName: \"kubernetes.io/projected/b367a50a-f9ee-4f39-896b-6baf0615bb8b-kube-api-access-dblxx\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.694999 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.694920 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b367a50a-f9ee-4f39-896b-6baf0615bb8b-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bvwbz\" (UID: \"b367a50a-f9ee-4f39-896b-6baf0615bb8b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:58.751973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:58.751943 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:26:59.062257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:59.062232 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz"] Oct 30 01:26:59.062799 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:26:59.062770 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb367a50a_f9ee_4f39_896b_6baf0615bb8b.slice/crio-7b467a498e019e717d51d0c21bb72697be03828f18bff789574972955bce0aae WatchSource:0}: Error finding container 7b467a498e019e717d51d0c21bb72697be03828f18bff789574972955bce0aae: Status 404 returned error can't find the container with id 7b467a498e019e717d51d0c21bb72697be03828f18bff789574972955bce0aae Oct 30 01:26:59.064822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:59.064789 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:26:59.064892 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:59.064853 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:26:59.064892 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:59.064883 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:26:59.828018 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:59.827986 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" event={"ID":"b367a50a-f9ee-4f39-896b-6baf0615bb8b","Type":"ContainerStarted","Data":"96f277cbe20f988696354d11d1420cc7bb45b034c24cc2a99685881ea04e3a0e"} Oct 30 01:26:59.828449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:59.828030 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" event={"ID":"b367a50a-f9ee-4f39-896b-6baf0615bb8b","Type":"ContainerStarted","Data":"7b467a498e019e717d51d0c21bb72697be03828f18bff789574972955bce0aae"} Oct 30 01:26:59.895138 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:26:59.895085 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" podStartSLOduration=1.895070029 podStartE2EDuration="1.895070029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:59.881869683 +0000 UTC m=+587.202826724" watchObservedRunningTime="2026-10-30 01:26:59.895070029 +0000 UTC m=+587.216027123" Oct 30 01:27:00.752625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:00.752592 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:27:00.754304 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:00.754278 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bvwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 01:27:00.754462 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:00.754342 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" podUID="b367a50a-f9ee-4f39-896b-6baf0615bb8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 01:27:01.753274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:01.753236 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bvwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 01:27:01.753721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:01.753294 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" podUID="b367a50a-f9ee-4f39-896b-6baf0615bb8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 01:27:02.752925 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:02.752884 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bvwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 01:27:02.753093 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:02.752966 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" podUID="b367a50a-f9ee-4f39-896b-6baf0615bb8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 01:27:03.752759 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:03.752718 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bvwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 01:27:03.753275 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:03.752799 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" podUID="b367a50a-f9ee-4f39-896b-6baf0615bb8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 01:27:04.752880 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:04.752845 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bvwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 01:27:04.753266 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:04.752912 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" podUID="b367a50a-f9ee-4f39-896b-6baf0615bb8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 01:27:05.752593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:05.752552 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bvwbz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 01:27:05.752810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:05.752605 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" podUID="b367a50a-f9ee-4f39-896b-6baf0615bb8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 01:27:07.186582 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:07.186549 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:27:07.187086 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:07.186854 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:27:07.187904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:07.187884 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bvwbz" Oct 30 01:27:12.867459 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:12.867423 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:27:12.869401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:12.869379 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:27:13.474173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.474139 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5"] Oct 30 01:27:13.478924 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.478902 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.515153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.514562 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5"] Oct 30 01:27:13.515153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.515013 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.515153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.515046 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5glgs\" (UniqueName: \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-kube-api-access-5glgs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.515442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.515177 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.515442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.515216 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.515442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.515257 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.515442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.515290 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.515442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.515353 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.515442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.515383 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.515442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.515418 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616114 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616080 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616295 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616128 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616295 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616253 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616301 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616361 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616395 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616522 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616553 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5glgs\" (UniqueName: \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-kube-api-access-5glgs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616613 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616662 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616676 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.616844 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.616761 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.617202 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.617173 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.617307 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.617245 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.618703 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.618683 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.619017 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.618996 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.636593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.636568 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.636840 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.636822 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5glgs\" (UniqueName: \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-kube-api-access-5glgs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-c56z5\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:13.792401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:13.792287 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:14.107110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:14.107078 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5"] Oct 30 01:27:14.107492 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:27:14.107294 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2206e13b_cd58_44c1_ae6d_964bbe9b8d5e.slice/crio-8a713ced9d689625d87f10076caf94ea5e639038e16a7948380baafa8a55c94d WatchSource:0}: Error finding container 8a713ced9d689625d87f10076caf94ea5e639038e16a7948380baafa8a55c94d: Status 404 returned error can't find the container with id 8a713ced9d689625d87f10076caf94ea5e639038e16a7948380baafa8a55c94d Oct 30 01:27:14.109579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:14.109545 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:27:14.109662 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:14.109611 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:27:14.109662 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:14.109639 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:27:14.899935 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:14.899902 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" event={"ID":"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e","Type":"ContainerStarted","Data":"f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279"} Oct 30 01:27:14.900107 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:14.899954 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" event={"ID":"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e","Type":"ContainerStarted","Data":"8a713ced9d689625d87f10076caf94ea5e639038e16a7948380baafa8a55c94d"} Oct 30 01:27:14.955724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:14.955675 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" podStartSLOduration=1.955660685 podStartE2EDuration="1.955660685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:14.95403772 +0000 UTC m=+602.274994757" watchObservedRunningTime="2026-10-30 01:27:14.955660685 +0000 UTC m=+602.276617725" Oct 30 01:27:15.792682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:15.792640 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:15.794379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:15.794352 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 01:27:15.794452 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:15.794407 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 01:27:16.792866 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:16.792833 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 01:27:16.793275 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:16.792887 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 01:27:17.792704 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:17.792674 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 01:27:17.792886 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:17.792733 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 01:27:18.794732 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:18.794692 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 01:27:18.795407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:18.795370 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 01:27:19.219751 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:27:19.219718 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:27:19.792993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:19.792954 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 01:27:19.793166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:19.793022 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 01:27:20.792716 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:20.792677 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c56z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 01:27:20.793187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:20.792746 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 01:27:22.536126 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:22.536088 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:22.536583 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:22.536503 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:22.537463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:22.537445 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:27:24.889052 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:24.889017 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd"] Oct 30 01:27:24.892785 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:24.892762 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:27:24.912971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:24.912944 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd"] Oct 30 01:27:24.955917 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:24.955880 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx8fj\" (UniqueName: \"kubernetes.io/projected/95d0a2a8-12be-4714-a14b-4bd6c253efb7-kube-api-access-cx8fj\") pod \"payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd\" (UID: \"95d0a2a8-12be-4714-a14b-4bd6c253efb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:27:24.956102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:24.956018 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d0a2a8-12be-4714-a14b-4bd6c253efb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd\" (UID: \"95d0a2a8-12be-4714-a14b-4bd6c253efb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:27:24.956355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:24.956312 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5b724cd8\"" Oct 30 01:27:24.956355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:24.956312 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5b724cd8-dockercfg-wng9n\"" Oct 30 01:27:25.057535 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.057502 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx8fj\" (UniqueName: \"kubernetes.io/projected/95d0a2a8-12be-4714-a14b-4bd6c253efb7-kube-api-access-cx8fj\") pod \"payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd\" (UID: \"95d0a2a8-12be-4714-a14b-4bd6c253efb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:27:25.057737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.057575 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d0a2a8-12be-4714-a14b-4bd6c253efb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd\" (UID: \"95d0a2a8-12be-4714-a14b-4bd6c253efb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:27:25.058217 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.058195 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d0a2a8-12be-4714-a14b-4bd6c253efb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd\" (UID: \"95d0a2a8-12be-4714-a14b-4bd6c253efb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:27:25.079146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.079122 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx8fj\" (UniqueName: \"kubernetes.io/projected/95d0a2a8-12be-4714-a14b-4bd6c253efb7-kube-api-access-cx8fj\") pod \"payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd\" (UID: \"95d0a2a8-12be-4714-a14b-4bd6c253efb7\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:27:25.206230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.206198 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:27:25.512211 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.512177 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd"] Oct 30 01:27:25.515514 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.515488 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:27:25.781690 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.781609 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g"] Oct 30 01:27:25.787221 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.787195 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:27:25.805523 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.805498 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g"] Oct 30 01:27:25.851221 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.851179 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f29772b9\"" Oct 30 01:27:25.851424 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.851399 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f29772b9-dockercfg-vbhvc\"" Oct 30 01:27:25.865076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.865040 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314ef409-5c59-4abb-9d47-46a916fae95f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g\" (UID: \"314ef409-5c59-4abb-9d47-46a916fae95f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:27:25.865256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.865209 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5cgn\" (UniqueName: \"kubernetes.io/projected/314ef409-5c59-4abb-9d47-46a916fae95f-kube-api-access-t5cgn\") pod \"payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g\" (UID: \"314ef409-5c59-4abb-9d47-46a916fae95f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:27:25.961865 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.961832 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" event={"ID":"95d0a2a8-12be-4714-a14b-4bd6c253efb7","Type":"ContainerStarted","Data":"8d17a48ea2f2e2da8ea3abe63b3599f1c8397b1b896a3df08b6294bccf6fd30f"} Oct 30 01:27:25.966407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.966379 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314ef409-5c59-4abb-9d47-46a916fae95f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g\" (UID: \"314ef409-5c59-4abb-9d47-46a916fae95f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:27:25.966536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.966437 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5cgn\" (UniqueName: \"kubernetes.io/projected/314ef409-5c59-4abb-9d47-46a916fae95f-kube-api-access-t5cgn\") pod \"payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g\" (UID: \"314ef409-5c59-4abb-9d47-46a916fae95f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:27:25.967241 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.967220 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314ef409-5c59-4abb-9d47-46a916fae95f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g\" (UID: \"314ef409-5c59-4abb-9d47-46a916fae95f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:27:25.986306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:25.986281 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5cgn\" (UniqueName: \"kubernetes.io/projected/314ef409-5c59-4abb-9d47-46a916fae95f-kube-api-access-t5cgn\") pod \"payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g\" (UID: \"314ef409-5c59-4abb-9d47-46a916fae95f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:27:26.102057 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.101560 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:27:26.427251 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.418973 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g"] Oct 30 01:27:26.427251 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:27:26.423301 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314ef409_5c59_4abb_9d47_46a916fae95f.slice/crio-98d219d4795dbe7ec37b9b6eb7432d74b610c47ad84a911c6694d51c3c501448 WatchSource:0}: Error finding container 98d219d4795dbe7ec37b9b6eb7432d74b610c47ad84a911c6694d51c3c501448: Status 404 returned error can't find the container with id 98d219d4795dbe7ec37b9b6eb7432d74b610c47ad84a911c6694d51c3c501448 Oct 30 01:27:26.808049 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.808012 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v"] Oct 30 01:27:26.816357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.815451 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:26.843501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.841163 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-18c05df8-openshift-default-dockercfg-75bz9\"" Oct 30 01:27:26.843501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.843083 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v"] Oct 30 01:27:26.878291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.877003 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-certs\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:26.878291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.877108 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-socket\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:26.878291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.877134 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-data\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:26.878291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.877173 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sclm9\" (UniqueName: \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-kube-api-access-sclm9\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:26.878291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.877207 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istiod-ca-cert\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:26.878291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.877278 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-podinfo\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:26.878291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.877315 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-credential-socket\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:26.878291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.877373 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-token\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:26.878291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.877404 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-envoy\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.989880 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-socket\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.989932 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-data\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.989977 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sclm9\" (UniqueName: \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-kube-api-access-sclm9\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.990009 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istiod-ca-cert\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.990088 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-podinfo\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.990128 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-credential-socket\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.990168 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-token\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.990204 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-envoy\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.990236 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-certs\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.990676 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-certs\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.990918 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-socket\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.991142 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-data\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.992196 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istiod-ca-cert\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.999094 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-podinfo\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.004960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:26.999376 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-credential-socket\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.012171 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.010088 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-envoy\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.014871 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.014843 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sclm9\" (UniqueName: \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-kube-api-access-sclm9\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.015711 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.015630 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" event={"ID":"314ef409-5c59-4abb-9d47-46a916fae95f","Type":"ContainerStarted","Data":"98d219d4795dbe7ec37b9b6eb7432d74b610c47ad84a911c6694d51c3c501448"} Oct 30 01:27:27.019540 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.019490 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-token\") pod \"e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.149108 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.149031 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:27.517489 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.517462 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v"] Oct 30 01:27:27.523109 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.523078 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:27:27.523370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.523349 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:27:27.523526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.523503 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:27:27.926810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.926727 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp"] Oct 30 01:27:27.934984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.934956 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:27.950605 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.950519 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp"] Oct 30 01:27:27.952451 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:27.952428 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f19ba569-dockercfg-xk4zx\"" Oct 30 01:27:28.025466 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.025417 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" event={"ID":"b4fc8e9d-801f-435a-8cb5-556a8af10d93","Type":"ContainerStarted","Data":"8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541"} Oct 30 01:27:28.027303 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.026413 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" event={"ID":"b4fc8e9d-801f-435a-8cb5-556a8af10d93","Type":"ContainerStarted","Data":"3fd788a750903d4c04588d71953cf2a84ab583b3c79d272d54e165a80c078510"} Oct 30 01:27:28.027303 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.027029 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-plugins-config-volume\") pod \"payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp\" (UID: \"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6\") " pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:28.027303 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.027123 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6p4p\" (UniqueName: \"kubernetes.io/projected/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-kube-api-access-t6p4p\") pod \"payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp\" (UID: \"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6\") " pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:28.057425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.057393 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f19ba569\"" Oct 30 01:27:28.128506 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.127918 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-plugins-config-volume\") pod \"payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp\" (UID: \"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6\") " pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:28.128506 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.128047 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6p4p\" (UniqueName: \"kubernetes.io/projected/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-kube-api-access-t6p4p\") pod \"payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp\" (UID: \"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6\") " pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:28.128965 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.128932 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-plugins-config-volume\") pod \"payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp\" (UID: \"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6\") " pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:28.149618 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.149597 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:28.150564 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.150540 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6p4p\" (UniqueName: \"kubernetes.io/projected/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-kube-api-access-t6p4p\") pod \"payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp\" (UID: \"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6\") " pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:28.155020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.154978 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:28.211428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.204253 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" podStartSLOduration=2.204233133 podStartE2EDuration="2.204233133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:28.090076144 +0000 UTC m=+615.411033184" watchObservedRunningTime="2026-10-30 01:27:28.204233133 +0000 UTC m=+615.525190175" Oct 30 01:27:28.275300 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.275254 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:28.585412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:28.585383 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp"] Oct 30 01:27:28.585707 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:27:28.585682 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ad004e7_5bdc_4207_a3e8_fe61ec450fa6.slice/crio-b006e759bf556c18388a39ef1dac5a767b0af3013e707bf0c8315e96f7090e24 WatchSource:0}: Error finding container b006e759bf556c18388a39ef1dac5a767b0af3013e707bf0c8315e96f7090e24: Status 404 returned error can't find the container with id b006e759bf556c18388a39ef1dac5a767b0af3013e707bf0c8315e96f7090e24 Oct 30 01:27:29.031892 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.031859 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" event={"ID":"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6","Type":"ContainerStarted","Data":"4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee"} Oct 30 01:27:29.032397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.031913 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" event={"ID":"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6","Type":"ContainerStarted","Data":"b006e759bf556c18388a39ef1dac5a767b0af3013e707bf0c8315e96f7090e24"} Oct 30 01:27:29.032397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.031935 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:29.033707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.033683 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" event={"ID":"95d0a2a8-12be-4714-a14b-4bd6c253efb7","Type":"ContainerStarted","Data":"abbe8065eb566fb9706f551459fd0186ffc2a42c5b56be633f9e6bcda3c4dd5f"} Oct 30 01:27:29.033805 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.033780 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:27:29.035225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.035203 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" event={"ID":"314ef409-5c59-4abb-9d47-46a916fae95f","Type":"ContainerStarted","Data":"f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890"} Oct 30 01:27:29.035398 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.035386 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:29.035480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.035409 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:27:29.036265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.036250 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:29.075624 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.075557 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" podStartSLOduration=2.07553743 podStartE2EDuration="2.07553743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:29.073766066 +0000 UTC m=+616.394723107" watchObservedRunningTime="2026-10-30 01:27:29.07553743 +0000 UTC m=+616.396494471" Oct 30 01:27:29.115607 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.115555 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" podStartSLOduration=2.179219394 podStartE2EDuration="5.115540358s" totalImagesPullingTime="2.936320964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:29.114573445 +0000 UTC m=+616.435530487" watchObservedRunningTime="2026-10-30 01:27:29.115540358 +0000 UTC m=+616.436497399" Oct 30 01:27:29.163049 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:29.163002 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" podStartSLOduration=2.103703489 podStartE2EDuration="4.162989693s" totalImagesPullingTime="2.059286204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:29.160892392 +0000 UTC m=+616.481849433" watchObservedRunningTime="2026-10-30 01:27:29.162989693 +0000 UTC m=+616.483946727" Oct 30 01:27:36.752028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:36.751997 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8"] Oct 30 01:27:36.755916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:36.755899 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:36.783314 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:36.783265 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:27:36.783447 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:36.783351 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:27:36.785777 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:36.785758 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8"] Oct 30 01:27:36.916913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:36.916881 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qcs7\" (UniqueName: \"kubernetes.io/projected/16b3b145-e4d7-4288-a9d5-2cce118a8267-kube-api-access-2qcs7\") pod \"maas-api-revoke-keys-e2e-migrate-d9801220-dldm8\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:36.917080 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:36.916922 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9801220-dldm8\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:36.917080 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:36.917054 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9801220-dldm8\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:37.018154 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:37.018062 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9801220-dldm8\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:37.018370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:37.018165 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qcs7\" (UniqueName: \"kubernetes.io/projected/16b3b145-e4d7-4288-a9d5-2cce118a8267-kube-api-access-2qcs7\") pod \"maas-api-revoke-keys-e2e-migrate-d9801220-dldm8\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:37.018370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:37.018203 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9801220-dldm8\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:37.019028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:37.018951 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9801220-dldm8\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:37.020851 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:37.020825 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d9801220-dldm8\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:37.037403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:37.037375 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qcs7\" (UniqueName: \"kubernetes.io/projected/16b3b145-e4d7-4288-a9d5-2cce118a8267-kube-api-access-2qcs7\") pod \"maas-api-revoke-keys-e2e-migrate-d9801220-dldm8\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:37.069312 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:37.069292 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:37.367642 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:37.367609 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8"] Oct 30 01:27:37.367923 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:27:37.367902 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b3b145_e4d7_4288_a9d5_2cce118a8267.slice/crio-211b1842dce557f3e27f557b8c1ba37f28a004c14b7825155f6495b1f8027bf2 WatchSource:0}: Error finding container 211b1842dce557f3e27f557b8c1ba37f28a004c14b7825155f6495b1f8027bf2: Status 404 returned error can't find the container with id 211b1842dce557f3e27f557b8c1ba37f28a004c14b7825155f6495b1f8027bf2 Oct 30 01:27:38.083693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:38.083656 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" event={"ID":"16b3b145-e4d7-4288-a9d5-2cce118a8267","Type":"ContainerStarted","Data":"211b1842dce557f3e27f557b8c1ba37f28a004c14b7825155f6495b1f8027bf2"} Oct 30 01:27:40.044446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:40.044317 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:27:40.044934 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:40.044547 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:27:40.044988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:40.044954 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:41.103689 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:41.103602 5055 generic.go:356] "Generic (PLEG): container finished" podID="16b3b145-e4d7-4288-a9d5-2cce118a8267" containerID="5b51e0e6d82e64f74d90ffc8ab75a70088b26bceb81b801725fa91a7e8b6d36e" exitCode=0 Oct 30 01:27:41.104076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:41.103697 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" event={"ID":"16b3b145-e4d7-4288-a9d5-2cce118a8267","Type":"ContainerDied","Data":"5b51e0e6d82e64f74d90ffc8ab75a70088b26bceb81b801725fa91a7e8b6d36e"} Oct 30 01:27:42.288642 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:42.288615 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:42.309510 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:42.309479 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-xks-ca\") pod \"16b3b145-e4d7-4288-a9d5-2cce118a8267\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " Oct 30 01:27:42.309646 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:42.309631 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-openshift-ca\") pod \"16b3b145-e4d7-4288-a9d5-2cce118a8267\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " Oct 30 01:27:42.309707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:42.309694 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16b3b145-e4d7-4288-a9d5-2cce118a8267-kube-api-access-2qcs7\" (UniqueName: \"kubernetes.io/projected/16b3b145-e4d7-4288-a9d5-2cce118a8267-kube-api-access-2qcs7\") pod \"16b3b145-e4d7-4288-a9d5-2cce118a8267\" (UID: \"16b3b145-e4d7-4288-a9d5-2cce118a8267\") " Oct 30 01:27:42.309982 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:42.309957 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-openshift-ca" pod "16b3b145-e4d7-4288-a9d5-2cce118a8267" (UID: "16b3b145-e4d7-4288-a9d5-2cce118a8267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:42.310102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:42.310088 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-openshift-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:42.312534 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:42.312509 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-xks-ca" pod "16b3b145-e4d7-4288-a9d5-2cce118a8267" (UID: "16b3b145-e4d7-4288-a9d5-2cce118a8267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:42.312534 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:42.312527 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16b3b145-e4d7-4288-a9d5-2cce118a8267-kube-api-access-2qcs7" pod "16b3b145-e4d7-4288-a9d5-2cce118a8267" (UID: "16b3b145-e4d7-4288-a9d5-2cce118a8267"). InnerVolumeSpecName "kube-api-access-2qcs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:42.411196 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:42.411120 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qcs7\" (UniqueName: \"kubernetes.io/projected/16b3b145-e4d7-4288-a9d5-2cce118a8267-kube-api-access-2qcs7\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:42.411196 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:42.411147 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16b3b145-e4d7-4288-a9d5-2cce118a8267-maas-api-xks-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:43.117577 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.117482 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" event={"ID":"16b3b145-e4d7-4288-a9d5-2cce118a8267","Type":"ContainerDied","Data":"211b1842dce557f3e27f557b8c1ba37f28a004c14b7825155f6495b1f8027bf2"} Oct 30 01:27:43.117577 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.117545 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="211b1842dce557f3e27f557b8c1ba37f28a004c14b7825155f6495b1f8027bf2" Oct 30 01:27:43.117793 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.117605 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8" Oct 30 01:27:43.482354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.480044 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5"] Oct 30 01:27:43.492403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.491959 5055 cpu_manager.go:414] "containerMap: removing container" podUID="16b3b145-e4d7-4288-a9d5-2cce118a8267" containerName="revoke-keys" Oct 30 01:27:43.492403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.491993 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16b3b145-e4d7-4288-a9d5-2cce118a8267" containerName="revoke-keys" Oct 30 01:27:43.492403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.492147 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="16b3b145-e4d7-4288-a9d5-2cce118a8267" containerName="revoke-keys" Oct 30 01:27:43.495908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.495882 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:43.500663 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.500641 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5"] Oct 30 01:27:43.511012 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.510964 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:27:43.511197 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.511178 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:27:43.525827 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.525787 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wnzl\" (UniqueName: \"kubernetes.io/projected/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-kube-api-access-5wnzl\") pod \"maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:43.526022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.525844 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:43.526022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.525963 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:43.626611 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.626573 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wnzl\" (UniqueName: \"kubernetes.io/projected/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-kube-api-access-5wnzl\") pod \"maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:43.626611 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.626618 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:43.626866 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.626638 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:43.627556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.627529 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:43.629255 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.629222 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:43.646914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.646882 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wnzl\" (UniqueName: \"kubernetes.io/projected/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-kube-api-access-5wnzl\") pod \"maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:43.813041 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:43.812971 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:44.135550 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:27:44.135520 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae86cf4_74a1_4f27_b0da_e04b5ef6fd01.slice/crio-2eb1704f8a7f0758a6c7793f8c2f3eb7e59e10aaf9e06726b80c20179e293759 WatchSource:0}: Error finding container 2eb1704f8a7f0758a6c7793f8c2f3eb7e59e10aaf9e06726b80c20179e293759: Status 404 returned error can't find the container with id 2eb1704f8a7f0758a6c7793f8c2f3eb7e59e10aaf9e06726b80c20179e293759 Oct 30 01:27:44.135550 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:44.135554 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5"] Oct 30 01:27:45.130354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:45.130292 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" event={"ID":"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01","Type":"ContainerStarted","Data":"704a485ef7351f302d5e092303d8823fcb2f5769a410d1a5832139bf382ea13b"} Oct 30 01:27:45.130720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:45.130379 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" event={"ID":"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01","Type":"ContainerStarted","Data":"2eb1704f8a7f0758a6c7793f8c2f3eb7e59e10aaf9e06726b80c20179e293759"} Oct 30 01:27:45.189351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:45.189063 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" podStartSLOduration=2.18904074 podStartE2EDuration="2.18904074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:45.170731553 +0000 UTC m=+632.491688594" watchObservedRunningTime="2026-10-30 01:27:45.18904074 +0000 UTC m=+632.509997781" Oct 30 01:27:46.137533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:46.137492 5055 generic.go:356] "Generic (PLEG): container finished" podID="6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01" containerID="704a485ef7351f302d5e092303d8823fcb2f5769a410d1a5832139bf382ea13b" exitCode=0 Oct 30 01:27:46.137913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:46.137579 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" event={"ID":"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01","Type":"ContainerDied","Data":"704a485ef7351f302d5e092303d8823fcb2f5769a410d1a5832139bf382ea13b"} Oct 30 01:27:47.402230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.402157 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:47.492772 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.492735 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-openshift-ca\") pod \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " Oct 30 01:27:47.492981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.492845 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-xks-ca\") pod \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " Oct 30 01:27:47.492981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.492903 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-kube-api-access-5wnzl\" (UniqueName: \"kubernetes.io/projected/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-kube-api-access-5wnzl\") pod \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\" (UID: \"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01\") " Oct 30 01:27:47.493551 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.493165 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-openshift-ca" pod "6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01" (UID: "6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:47.496358 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.495280 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-kube-api-access-5wnzl" pod "6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01" (UID: "6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01"). InnerVolumeSpecName "kube-api-access-5wnzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:47.496358 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.495773 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-xks-ca" pod "6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01" (UID: "6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:47.595062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.595019 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-openshift-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:47.595062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.595062 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-maas-api-xks-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:47.595297 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.595076 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wnzl\" (UniqueName: \"kubernetes.io/projected/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01-kube-api-access-5wnzl\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:47.615417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.614827 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk"] Oct 30 01:27:47.617548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.615832 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01" containerName="revoke-keys" Oct 30 01:27:47.617548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.615860 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01" containerName="revoke-keys" Oct 30 01:27:47.617548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.616059 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01" containerName="revoke-keys" Oct 30 01:27:47.625614 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.623793 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:47.645990 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.645958 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk"] Oct 30 01:27:47.696735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.696485 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzst\" (UniqueName: \"kubernetes.io/projected/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-kube-api-access-qjzst\") pod \"maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:47.696735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.696532 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:47.696735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.696583 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:47.798122 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.798086 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzst\" (UniqueName: \"kubernetes.io/projected/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-kube-api-access-qjzst\") pod \"maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:47.798312 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.798138 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:47.798312 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.798190 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:47.798886 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.798846 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:47.800743 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.800723 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:47.817406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.817379 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzst\" (UniqueName: \"kubernetes.io/projected/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-kube-api-access-qjzst\") pod \"maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:47.941243 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:47.941212 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:48.149568 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:48.149544 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" Oct 30 01:27:48.149730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:48.149543 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5" event={"ID":"6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01","Type":"ContainerDied","Data":"2eb1704f8a7f0758a6c7793f8c2f3eb7e59e10aaf9e06726b80c20179e293759"} Oct 30 01:27:48.149730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:48.149661 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb1704f8a7f0758a6c7793f8c2f3eb7e59e10aaf9e06726b80c20179e293759" Oct 30 01:27:48.289202 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:48.289111 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk"] Oct 30 01:27:48.293523 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:27:48.293490 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74a047d0_c3fd_4617_8d6a_ca34a5981d7f.slice/crio-9969a24f9e437515d526adb4bb7d50cf95c2311c0446560703607abaff3844f7 WatchSource:0}: Error finding container 9969a24f9e437515d526adb4bb7d50cf95c2311c0446560703607abaff3844f7: Status 404 returned error can't find the container with id 9969a24f9e437515d526adb4bb7d50cf95c2311c0446560703607abaff3844f7 Oct 30 01:27:49.157141 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:49.157105 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" event={"ID":"74a047d0-c3fd-4617-8d6a-ca34a5981d7f","Type":"ContainerStarted","Data":"28a92db128cddcda71a44757b3e0039cf3e86c696e6fd6f1d2ceeea9f7bc2f50"} Oct 30 01:27:49.157586 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:49.157164 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" event={"ID":"74a047d0-c3fd-4617-8d6a-ca34a5981d7f","Type":"ContainerStarted","Data":"9969a24f9e437515d526adb4bb7d50cf95c2311c0446560703607abaff3844f7"} Oct 30 01:27:49.199460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:49.199406 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" podStartSLOduration=2.199386511 podStartE2EDuration="2.199386511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:49.197024451 +0000 UTC m=+636.517981484" watchObservedRunningTime="2026-10-30 01:27:49.199386511 +0000 UTC m=+636.520343551" Oct 30 01:27:49.220144 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:27:49.220111 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:27:50.163938 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:50.163896 5055 generic.go:356] "Generic (PLEG): container finished" podID="74a047d0-c3fd-4617-8d6a-ca34a5981d7f" containerID="28a92db128cddcda71a44757b3e0039cf3e86c696e6fd6f1d2ceeea9f7bc2f50" exitCode=0 Oct 30 01:27:50.164394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:50.163986 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" event={"ID":"74a047d0-c3fd-4617-8d6a-ca34a5981d7f","Type":"ContainerDied","Data":"28a92db128cddcda71a44757b3e0039cf3e86c696e6fd6f1d2ceeea9f7bc2f50"} Oct 30 01:27:51.307693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:51.307666 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:51.440014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:51.439936 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-openshift-ca\") pod \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " Oct 30 01:27:51.440162 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:51.440057 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-xks-ca\") pod \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " Oct 30 01:27:51.440162 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:51.440121 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-kube-api-access-qjzst\" (UniqueName: \"kubernetes.io/projected/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-kube-api-access-qjzst\") pod \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\" (UID: \"74a047d0-c3fd-4617-8d6a-ca34a5981d7f\") " Oct 30 01:27:51.440432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:51.440400 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-openshift-ca" pod "74a047d0-c3fd-4617-8d6a-ca34a5981d7f" (UID: "74a047d0-c3fd-4617-8d6a-ca34a5981d7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:51.442379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:51.442346 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-xks-ca" pod "74a047d0-c3fd-4617-8d6a-ca34a5981d7f" (UID: "74a047d0-c3fd-4617-8d6a-ca34a5981d7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:51.442503 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:51.442380 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-kube-api-access-qjzst" pod "74a047d0-c3fd-4617-8d6a-ca34a5981d7f" (UID: "74a047d0-c3fd-4617-8d6a-ca34a5981d7f"). InnerVolumeSpecName "kube-api-access-qjzst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:51.541737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:51.541708 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjzst\" (UniqueName: \"kubernetes.io/projected/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-kube-api-access-qjzst\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:51.541737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:51.541734 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-openshift-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:51.541737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:51.541743 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a047d0-c3fd-4617-8d6a-ca34a5981d7f-maas-api-xks-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:52.175886 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:52.175848 5055 generic.go:356] "Generic (PLEG): container finished" podID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerID="8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541" exitCode=0 Oct 30 01:27:52.176091 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:52.175920 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" event={"ID":"b4fc8e9d-801f-435a-8cb5-556a8af10d93","Type":"ContainerDied","Data":"8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541"} Oct 30 01:27:52.176391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:52.176375 5055 scope.go:122] "RemoveContainer" containerID="8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541" Oct 30 01:27:52.177369 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:52.177317 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" Oct 30 01:27:52.177369 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:52.177356 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk" event={"ID":"74a047d0-c3fd-4617-8d6a-ca34a5981d7f","Type":"ContainerDied","Data":"9969a24f9e437515d526adb4bb7d50cf95c2311c0446560703607abaff3844f7"} Oct 30 01:27:52.177525 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:52.177397 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9969a24f9e437515d526adb4bb7d50cf95c2311c0446560703607abaff3844f7" Oct 30 01:27:52.177734 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:52.177687 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:27:52.177801 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:52.177789 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:27:52.177844 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:52.177833 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:27:53.192657 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:53.192615 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" event={"ID":"b4fc8e9d-801f-435a-8cb5-556a8af10d93","Type":"ContainerStarted","Data":"832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa"} Oct 30 01:27:53.919228 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:53.919186 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp"] Oct 30 01:27:54.149873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:54.149828 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:54.151314 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:54.151277 5055 patch_prober.go:29] interesting pod/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 01:27:54.151523 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:54.151385 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 01:27:54.899643 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:54.899606 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8"] Oct 30 01:27:54.902873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:54.902848 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d9801220-dldm8"] Oct 30 01:27:55.134768 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.134741 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg"] Oct 30 01:27:55.135244 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.135231 5055 cpu_manager.go:414] "containerMap: removing container" podUID="74a047d0-c3fd-4617-8d6a-ca34a5981d7f" containerName="revoke-keys" Oct 30 01:27:55.135286 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.135245 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a047d0-c3fd-4617-8d6a-ca34a5981d7f" containerName="revoke-keys" Oct 30 01:27:55.135336 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.135316 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a047d0-c3fd-4617-8d6a-ca34a5981d7f" containerName="revoke-keys" Oct 30 01:27:55.141636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.141615 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.149561 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.149543 5055 patch_prober.go:29] interesting pod/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 01:27:55.149635 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.149589 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 01:27:55.176409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.162156 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg"] Oct 30 01:27:55.224379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.215311 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:27:55.224620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.223709 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:27:55.231394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.228905 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rptsg\" (UniqueName: \"kubernetes.io/projected/6cb2c3dd-9010-4068-8d87-a662089c97a3-kube-api-access-rptsg\") pod \"maas-api-revoke-keys-e2e-derive-18c05df8-nshpg\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.231394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.228985 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-18c05df8-nshpg\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.231394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.229064 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-18c05df8-nshpg\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.329587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.329552 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-18c05df8-nshpg\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.329778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.329608 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rptsg\" (UniqueName: \"kubernetes.io/projected/6cb2c3dd-9010-4068-8d87-a662089c97a3-kube-api-access-rptsg\") pod \"maas-api-revoke-keys-e2e-derive-18c05df8-nshpg\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.329778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.329651 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-18c05df8-nshpg\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.330245 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.330227 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-18c05df8-nshpg\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.332081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.332057 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-18c05df8-nshpg\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.352869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.352845 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rptsg\" (UniqueName: \"kubernetes.io/projected/6cb2c3dd-9010-4068-8d87-a662089c97a3-kube-api-access-rptsg\") pod \"maas-api-revoke-keys-e2e-derive-18c05df8-nshpg\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.456752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.456723 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:55.803460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:55.803432 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg"] Oct 30 01:27:55.803937 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:27:55.803908 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb2c3dd_9010_4068_8d87_a662089c97a3.slice/crio-1223c0ca5270d240e53539c2cbe1a11f7b5083d5d0bb844f3d3b81b5a3878c54 WatchSource:0}: Error finding container 1223c0ca5270d240e53539c2cbe1a11f7b5083d5d0bb844f3d3b81b5a3878c54: Status 404 returned error can't find the container with id 1223c0ca5270d240e53539c2cbe1a11f7b5083d5d0bb844f3d3b81b5a3878c54 Oct 30 01:27:56.150227 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:56.150197 5055 patch_prober.go:29] interesting pod/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 01:27:56.150607 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:56.150251 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 01:27:56.241208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:56.241174 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" event={"ID":"6cb2c3dd-9010-4068-8d87-a662089c97a3","Type":"ContainerStarted","Data":"f289ba428e7741739aa4fd4b07972b76ff66b451252842d2efbe4b4ad647e7a4"} Oct 30 01:27:56.241386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:56.241262 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" event={"ID":"6cb2c3dd-9010-4068-8d87-a662089c97a3","Type":"ContainerStarted","Data":"1223c0ca5270d240e53539c2cbe1a11f7b5083d5d0bb844f3d3b81b5a3878c54"} Oct 30 01:27:56.820425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:56.820397 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16b3b145-e4d7-4288-a9d5-2cce118a8267" path="/var/lib/kubelet/pods/16b3b145-e4d7-4288-a9d5-2cce118a8267/volumes" Oct 30 01:27:57.149878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:57.149796 5055 patch_prober.go:29] interesting pod/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 01:27:57.150032 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:57.149877 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 01:27:57.248305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:57.248252 5055 generic.go:356] "Generic (PLEG): container finished" podID="6cb2c3dd-9010-4068-8d87-a662089c97a3" containerID="f289ba428e7741739aa4fd4b07972b76ff66b451252842d2efbe4b4ad647e7a4" exitCode=0 Oct 30 01:27:57.248708 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:57.248352 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" event={"ID":"6cb2c3dd-9010-4068-8d87-a662089c97a3","Type":"ContainerDied","Data":"f289ba428e7741739aa4fd4b07972b76ff66b451252842d2efbe4b4ad647e7a4"} Oct 30 01:27:58.070872 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.070840 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd"] Oct 30 01:27:58.149882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.149849 5055 patch_prober.go:29] interesting pod/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Oct 30 01:27:58.150062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.149920 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Oct 30 01:27:58.449141 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.449111 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:58.462981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.462948 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb2c3dd-9010-4068-8d87-a662089c97a3-kube-api-access-rptsg\" (UniqueName: \"kubernetes.io/projected/6cb2c3dd-9010-4068-8d87-a662089c97a3-kube-api-access-rptsg\") pod \"6cb2c3dd-9010-4068-8d87-a662089c97a3\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " Oct 30 01:27:58.463179 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.462989 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-xks-ca\") pod \"6cb2c3dd-9010-4068-8d87-a662089c97a3\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " Oct 30 01:27:58.463179 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.463078 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-openshift-ca\") pod \"6cb2c3dd-9010-4068-8d87-a662089c97a3\" (UID: \"6cb2c3dd-9010-4068-8d87-a662089c97a3\") " Oct 30 01:27:58.463633 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.463606 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-openshift-ca" pod "6cb2c3dd-9010-4068-8d87-a662089c97a3" (UID: "6cb2c3dd-9010-4068-8d87-a662089c97a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:58.465310 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.465285 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-xks-ca" pod "6cb2c3dd-9010-4068-8d87-a662089c97a3" (UID: "6cb2c3dd-9010-4068-8d87-a662089c97a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:58.465421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.465380 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb2c3dd-9010-4068-8d87-a662089c97a3-kube-api-access-rptsg" pod "6cb2c3dd-9010-4068-8d87-a662089c97a3" (UID: "6cb2c3dd-9010-4068-8d87-a662089c97a3"). InnerVolumeSpecName "kube-api-access-rptsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:58.564281 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.564249 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-openshift-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.564432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.564284 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rptsg\" (UniqueName: \"kubernetes.io/projected/6cb2c3dd-9010-4068-8d87-a662089c97a3-kube-api-access-rptsg\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.564432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.564300 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb2c3dd-9010-4068-8d87-a662089c97a3-maas-api-xks-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:58.789910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.789878 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g"] Oct 30 01:27:58.919758 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:58.919714 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" podUID="9ad004e7-5bdc-4207-a3e8-fe61ec450fa6" containerName="payload-processing" containerID="cri-o://4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee" gracePeriod=25 Oct 30 01:27:59.036293 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.036267 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:59.154265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.154232 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:59.195934 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.195904 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:59.261422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.261385 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" Oct 30 01:27:59.261422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.261402 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg" event={"ID":"6cb2c3dd-9010-4068-8d87-a662089c97a3","Type":"ContainerDied","Data":"1223c0ca5270d240e53539c2cbe1a11f7b5083d5d0bb844f3d3b81b5a3878c54"} Oct 30 01:27:59.261649 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.261461 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1223c0ca5270d240e53539c2cbe1a11f7b5083d5d0bb844f3d3b81b5a3878c54" Oct 30 01:27:59.272045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.272010 5055 generic.go:356] "Generic (PLEG): container finished" podID="9ad004e7-5bdc-4207-a3e8-fe61ec450fa6" containerID="4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee" exitCode=0 Oct 30 01:27:59.272888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.272872 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" Oct 30 01:27:59.273003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.272984 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" event={"ID":"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6","Type":"ContainerDied","Data":"4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee"} Oct 30 01:27:59.273050 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.273027 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp" event={"ID":"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6","Type":"ContainerDied","Data":"b006e759bf556c18388a39ef1dac5a767b0af3013e707bf0c8315e96f7090e24"} Oct 30 01:27:59.273094 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.273049 5055 scope.go:122] "RemoveContainer" containerID="4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee" Oct 30 01:27:59.274416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.274101 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:27:59.281873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.280287 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-plugins-config-volume\") pod \"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6\" (UID: \"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6\") " Oct 30 01:27:59.281873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.280387 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-kube-api-access-t6p4p\" (UniqueName: \"kubernetes.io/projected/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-kube-api-access-t6p4p\") pod \"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6\" (UID: \"9ad004e7-5bdc-4207-a3e8-fe61ec450fa6\") " Oct 30 01:27:59.281873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.281018 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-plugins-config-volume" pod "9ad004e7-5bdc-4207-a3e8-fe61ec450fa6" (UID: "9ad004e7-5bdc-4207-a3e8-fe61ec450fa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:59.296607 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.296552 5055 scope.go:122] "RemoveContainer" containerID="4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee" Oct 30 01:27:59.296917 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:27:59.296852 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee\": container with ID starting with 4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee not found: ID does not exist" containerID="4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee" Oct 30 01:27:59.296917 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.296891 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee"} err="failed to get container status \"4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee\": rpc error: code = NotFound desc = could not find container \"4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee\": container with ID starting with 4729ad665ae6d23b70a176343e9fb9f480c678f913fd83353624004b8b1eb7ee not found: ID does not exist" Oct 30 01:27:59.301174 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.301138 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-kube-api-access-t6p4p" pod "9ad004e7-5bdc-4207-a3e8-fe61ec450fa6" (UID: "9ad004e7-5bdc-4207-a3e8-fe61ec450fa6"). InnerVolumeSpecName "kube-api-access-t6p4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:59.382916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.382376 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-plugins-config-volume\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.382916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.382406 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6p4p\" (UniqueName: \"kubernetes.io/projected/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6-kube-api-access-t6p4p\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.422951 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.422921 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk"] Oct 30 01:27:59.425697 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.425675 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5b724cd8-jrttk"] Oct 30 01:27:59.618625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.618585 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp"] Oct 30 01:27:59.624396 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:27:59.624371 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f19ba569-54465db7d8-w5rwp"] Oct 30 01:28:00.095988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:00.095958 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5"] Oct 30 01:28:00.097839 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:00.097813 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f19ba569-mbcj5"] Oct 30 01:28:00.826678 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:00.826640 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01" path="/var/lib/kubelet/pods/6ae86cf4-74a1-4f27-b0da-e04b5ef6fd01/volumes" Oct 30 01:28:00.827284 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:00.827200 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a047d0-c3fd-4617-8d6a-ca34a5981d7f" path="/var/lib/kubelet/pods/74a047d0-c3fd-4617-8d6a-ca34a5981d7f/volumes" Oct 30 01:28:00.827695 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:00.827668 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ad004e7-5bdc-4207-a3e8-fe61ec450fa6" path="/var/lib/kubelet/pods/9ad004e7-5bdc-4207-a3e8-fe61ec450fa6/volumes" Oct 30 01:28:02.097052 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.097002 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt"] Oct 30 01:28:02.097781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.097760 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9ad004e7-5bdc-4207-a3e8-fe61ec450fa6" containerName="payload-processing" Oct 30 01:28:02.097781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.097782 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ad004e7-5bdc-4207-a3e8-fe61ec450fa6" containerName="payload-processing" Oct 30 01:28:02.097963 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.097795 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6cb2c3dd-9010-4068-8d87-a662089c97a3" containerName="revoke-keys" Oct 30 01:28:02.097963 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.097803 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb2c3dd-9010-4068-8d87-a662089c97a3" containerName="revoke-keys" Oct 30 01:28:02.097963 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.097932 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ad004e7-5bdc-4207-a3e8-fe61ec450fa6" containerName="payload-processing" Oct 30 01:28:02.097963 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.097946 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb2c3dd-9010-4068-8d87-a662089c97a3" containerName="revoke-keys" Oct 30 01:28:02.106981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.106955 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.125493 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.125002 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a3a2ac-openshift-default-dockercfg-mwwtj\"" Oct 30 01:28:02.140890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.140845 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt"] Oct 30 01:28:02.213949 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.213911 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.214144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.213972 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.214144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.214017 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.214144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.214047 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d144a49-a6b5-4282-b2af-b7945045301a-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.214144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.214082 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhtv\" (UniqueName: \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-kube-api-access-zfhtv\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.214144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.214121 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.214144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.214145 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.214576 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.214209 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d144a49-a6b5-4282-b2af-b7945045301a-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.214576 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.214238 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315042 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315015 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315251 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315057 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315251 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315079 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d144a49-a6b5-4282-b2af-b7945045301a-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315251 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315192 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhtv\" (UniqueName: \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-kube-api-access-zfhtv\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315429 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315251 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315429 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315277 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315429 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315368 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d144a49-a6b5-4282-b2af-b7945045301a-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315436 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315496 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315523 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.315930 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.315910 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.316054 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.316024 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.316110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.316077 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.316485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.316459 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d144a49-a6b5-4282-b2af-b7945045301a-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.317916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.317895 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d144a49-a6b5-4282-b2af-b7945045301a-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.318230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.318214 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.340438 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.340412 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhtv\" (UniqueName: \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-kube-api-access-zfhtv\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.342465 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.342441 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.427478 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.427394 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:02.743907 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:28:02.743875 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d144a49_a6b5_4282_b2af_b7945045301a.slice/crio-a64c17c6395433d766e26d8d6a45d847105ed8c4108bce30456c5388243a28f8 WatchSource:0}: Error finding container a64c17c6395433d766e26d8d6a45d847105ed8c4108bce30456c5388243a28f8: Status 404 returned error can't find the container with id a64c17c6395433d766e26d8d6a45d847105ed8c4108bce30456c5388243a28f8 Oct 30 01:28:02.744048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.744025 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt"] Oct 30 01:28:02.746489 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.746461 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:28:02.746599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.746524 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:28:02.746599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:02.746553 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:28:03.071908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.071607 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" podUID="95d0a2a8-12be-4714-a14b-4bd6c253efb7" containerName="payload-pre-processing" containerID="cri-o://abbe8065eb566fb9706f551459fd0186ffc2a42c5b56be633f9e6bcda3c4dd5f" gracePeriod=25 Oct 30 01:28:03.300056 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.300023 5055 generic.go:356] "Generic (PLEG): container finished" podID="95d0a2a8-12be-4714-a14b-4bd6c253efb7" containerID="abbe8065eb566fb9706f551459fd0186ffc2a42c5b56be633f9e6bcda3c4dd5f" exitCode=0 Oct 30 01:28:03.300586 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.300117 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" event={"ID":"95d0a2a8-12be-4714-a14b-4bd6c253efb7","Type":"ContainerDied","Data":"abbe8065eb566fb9706f551459fd0186ffc2a42c5b56be633f9e6bcda3c4dd5f"} Oct 30 01:28:03.301911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.301880 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" event={"ID":"0d144a49-a6b5-4282-b2af-b7945045301a","Type":"ContainerStarted","Data":"28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f"} Oct 30 01:28:03.302040 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.301939 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" event={"ID":"0d144a49-a6b5-4282-b2af-b7945045301a","Type":"ContainerStarted","Data":"a64c17c6395433d766e26d8d6a45d847105ed8c4108bce30456c5388243a28f8"} Oct 30 01:28:03.318682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.318662 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:28:03.369669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.369616 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" podStartSLOduration=1.369601243 podStartE2EDuration="1.369601243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:03.36922904 +0000 UTC m=+650.690186105" watchObservedRunningTime="2026-10-30 01:28:03.369601243 +0000 UTC m=+650.690558282" Oct 30 01:28:03.426318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.426289 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d0a2a8-12be-4714-a14b-4bd6c253efb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d0a2a8-12be-4714-a14b-4bd6c253efb7-plugins-config-volume\") pod \"95d0a2a8-12be-4714-a14b-4bd6c253efb7\" (UID: \"95d0a2a8-12be-4714-a14b-4bd6c253efb7\") " Oct 30 01:28:03.426495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.426407 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d0a2a8-12be-4714-a14b-4bd6c253efb7-kube-api-access-cx8fj\" (UniqueName: \"kubernetes.io/projected/95d0a2a8-12be-4714-a14b-4bd6c253efb7-kube-api-access-cx8fj\") pod \"95d0a2a8-12be-4714-a14b-4bd6c253efb7\" (UID: \"95d0a2a8-12be-4714-a14b-4bd6c253efb7\") " Oct 30 01:28:03.426609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.426591 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d0a2a8-12be-4714-a14b-4bd6c253efb7-plugins-config-volume" pod "95d0a2a8-12be-4714-a14b-4bd6c253efb7" (UID: "95d0a2a8-12be-4714-a14b-4bd6c253efb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:03.427311 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.427286 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d0a2a8-12be-4714-a14b-4bd6c253efb7-plugins-config-volume\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:03.427671 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.427650 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:03.429023 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.428987 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d0a2a8-12be-4714-a14b-4bd6c253efb7-kube-api-access-cx8fj" pod "95d0a2a8-12be-4714-a14b-4bd6c253efb7" (UID: "95d0a2a8-12be-4714-a14b-4bd6c253efb7"). InnerVolumeSpecName "kube-api-access-cx8fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:03.434229 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.434207 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:03.528087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.528045 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx8fj\" (UniqueName: \"kubernetes.io/projected/95d0a2a8-12be-4714-a14b-4bd6c253efb7-kube-api-access-cx8fj\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:03.793515 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:03.793452 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" podUID="314ef409-5c59-4abb-9d47-46a916fae95f" containerName="payload-pre-processing" containerID="cri-o://f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890" gracePeriod=25 Oct 30 01:28:04.119251 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.119221 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:28:04.235687 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.235656 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314ef409-5c59-4abb-9d47-46a916fae95f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314ef409-5c59-4abb-9d47-46a916fae95f-plugins-config-volume\") pod \"314ef409-5c59-4abb-9d47-46a916fae95f\" (UID: \"314ef409-5c59-4abb-9d47-46a916fae95f\") " Oct 30 01:28:04.235870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.235807 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314ef409-5c59-4abb-9d47-46a916fae95f-kube-api-access-t5cgn\" (UniqueName: \"kubernetes.io/projected/314ef409-5c59-4abb-9d47-46a916fae95f-kube-api-access-t5cgn\") pod \"314ef409-5c59-4abb-9d47-46a916fae95f\" (UID: \"314ef409-5c59-4abb-9d47-46a916fae95f\") " Oct 30 01:28:04.236063 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.236038 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314ef409-5c59-4abb-9d47-46a916fae95f-plugins-config-volume" pod "314ef409-5c59-4abb-9d47-46a916fae95f" (UID: "314ef409-5c59-4abb-9d47-46a916fae95f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:04.238113 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.238091 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314ef409-5c59-4abb-9d47-46a916fae95f-kube-api-access-t5cgn" pod "314ef409-5c59-4abb-9d47-46a916fae95f" (UID: "314ef409-5c59-4abb-9d47-46a916fae95f"). InnerVolumeSpecName "kube-api-access-t5cgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:04.309652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.309620 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" Oct 30 01:28:04.310104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.309648 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd" event={"ID":"95d0a2a8-12be-4714-a14b-4bd6c253efb7","Type":"ContainerDied","Data":"8d17a48ea2f2e2da8ea3abe63b3599f1c8397b1b896a3df08b6294bccf6fd30f"} Oct 30 01:28:04.310104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.309718 5055 scope.go:122] "RemoveContainer" containerID="abbe8065eb566fb9706f551459fd0186ffc2a42c5b56be633f9e6bcda3c4dd5f" Oct 30 01:28:04.311209 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.311182 5055 generic.go:356] "Generic (PLEG): container finished" podID="314ef409-5c59-4abb-9d47-46a916fae95f" containerID="f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890" exitCode=0 Oct 30 01:28:04.311365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.311226 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" Oct 30 01:28:04.311365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.311281 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" event={"ID":"314ef409-5c59-4abb-9d47-46a916fae95f","Type":"ContainerDied","Data":"f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890"} Oct 30 01:28:04.311365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.311312 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g" event={"ID":"314ef409-5c59-4abb-9d47-46a916fae95f","Type":"ContainerDied","Data":"98d219d4795dbe7ec37b9b6eb7432d74b610c47ad84a911c6694d51c3c501448"} Oct 30 01:28:04.312230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.311877 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:04.313095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.313071 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:04.319845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.319830 5055 scope.go:122] "RemoveContainer" containerID="f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890" Oct 30 01:28:04.333100 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.333081 5055 scope.go:122] "RemoveContainer" containerID="f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890" Oct 30 01:28:04.333394 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:28:04.333375 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890\": container with ID starting with f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890 not found: ID does not exist" containerID="f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890" Oct 30 01:28:04.333458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.333403 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890"} err="failed to get container status \"f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890\": rpc error: code = NotFound desc = could not find container \"f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890\": container with ID starting with f2a814abe6f1c82c4da0503f956e549154659a6739f09d509afef88029612890 not found: ID does not exist" Oct 30 01:28:04.336891 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.336870 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5cgn\" (UniqueName: \"kubernetes.io/projected/314ef409-5c59-4abb-9d47-46a916fae95f-kube-api-access-t5cgn\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:04.336891 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.336891 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/314ef409-5c59-4abb-9d47-46a916fae95f-plugins-config-volume\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:04.433269 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.433190 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd"] Oct 30 01:28:04.437088 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.437058 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5b724cd8-5575b5f68c-p4gtd"] Oct 30 01:28:04.471767 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.471735 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g"] Oct 30 01:28:04.473588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.473567 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-f29772b9-6d86d6b795-ck56g"] Oct 30 01:28:04.820983 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.820949 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314ef409-5c59-4abb-9d47-46a916fae95f" path="/var/lib/kubelet/pods/314ef409-5c59-4abb-9d47-46a916fae95f/volumes" Oct 30 01:28:04.821539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:04.821518 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d0a2a8-12be-4714-a14b-4bd6c253efb7" path="/var/lib/kubelet/pods/95d0a2a8-12be-4714-a14b-4bd6c253efb7/volumes" Oct 30 01:28:08.328421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:08.328383 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg"] Oct 30 01:28:08.340498 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:08.340468 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-18c05df8-nshpg"] Oct 30 01:28:08.499902 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:08.499866 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v"] Oct 30 01:28:08.500272 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:08.500120 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" containerID="cri-o://832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa" gracePeriod=30 Oct 30 01:28:08.820475 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:08.820441 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb2c3dd-9010-4068-8d87-a662089c97a3" path="/var/lib/kubelet/pods/6cb2c3dd-9010-4068-8d87-a662089c97a3/volumes" Oct 30 01:28:13.768857 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.768800 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:28:13.853593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.853566 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-envoy\") pod \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " Oct 30 01:28:13.853770 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.853661 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-token\" (UniqueName: \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-token\") pod \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " Oct 30 01:28:13.853770 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.853706 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-credential-socket\") pod \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " Oct 30 01:28:13.853770 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.853741 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istiod-ca-cert\") pod \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " Oct 30 01:28:13.853932 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.853843 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-kube-api-access-sclm9\" (UniqueName: \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-kube-api-access-sclm9\") pod \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " Oct 30 01:28:13.853932 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.853882 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-data\") pod \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " Oct 30 01:28:13.853932 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.853924 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-podinfo\") pod \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " Oct 30 01:28:13.854077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.853977 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-certs\") pod \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " Oct 30 01:28:13.854077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.853980 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-credential-socket" pod "b4fc8e9d-801f-435a-8cb5-556a8af10d93" (UID: "b4fc8e9d-801f-435a-8cb5-556a8af10d93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:13.854077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.854007 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-socket\") pod \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\" (UID: \"b4fc8e9d-801f-435a-8cb5-556a8af10d93\") " Oct 30 01:28:13.854228 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.854141 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-data" pod "b4fc8e9d-801f-435a-8cb5-556a8af10d93" (UID: "b4fc8e9d-801f-435a-8cb5-556a8af10d93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:13.854228 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.854163 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istiod-ca-cert" pod "b4fc8e9d-801f-435a-8cb5-556a8af10d93" (UID: "b4fc8e9d-801f-435a-8cb5-556a8af10d93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:13.854383 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.854303 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-socket" pod "b4fc8e9d-801f-435a-8cb5-556a8af10d93" (UID: "b4fc8e9d-801f-435a-8cb5-556a8af10d93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:13.854383 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.854356 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-certs" pod "b4fc8e9d-801f-435a-8cb5-556a8af10d93" (UID: "b4fc8e9d-801f-435a-8cb5-556a8af10d93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:13.854784 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.854728 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:13.854784 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.854750 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:13.854908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.854784 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:13.854908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.854798 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:13.854908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.854813 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:13.856950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.856841 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-token" pod "b4fc8e9d-801f-435a-8cb5-556a8af10d93" (UID: "b4fc8e9d-801f-435a-8cb5-556a8af10d93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:13.856950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.856867 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-podinfo" pod "b4fc8e9d-801f-435a-8cb5-556a8af10d93" (UID: "b4fc8e9d-801f-435a-8cb5-556a8af10d93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:13.857187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.857140 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-kube-api-access-sclm9" pod "b4fc8e9d-801f-435a-8cb5-556a8af10d93" (UID: "b4fc8e9d-801f-435a-8cb5-556a8af10d93"). InnerVolumeSpecName "kube-api-access-sclm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:13.857567 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.857542 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-envoy" pod "b4fc8e9d-801f-435a-8cb5-556a8af10d93" (UID: "b4fc8e9d-801f-435a-8cb5-556a8af10d93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:13.955688 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.955653 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:13.955688 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.955685 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sclm9\" (UniqueName: \"kubernetes.io/projected/b4fc8e9d-801f-435a-8cb5-556a8af10d93-kube-api-access-sclm9\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:13.955873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.955698 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:13.955873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:13.955709 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4fc8e9d-801f-435a-8cb5-556a8af10d93-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:28:14.387434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.387403 5055 generic.go:356] "Generic (PLEG): container finished" podID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerID="832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa" exitCode=0 Oct 30 01:28:14.387617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.387472 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" Oct 30 01:28:14.387617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.387498 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" event={"ID":"b4fc8e9d-801f-435a-8cb5-556a8af10d93","Type":"ContainerDied","Data":"832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa"} Oct 30 01:28:14.387617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.387563 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v" event={"ID":"b4fc8e9d-801f-435a-8cb5-556a8af10d93","Type":"ContainerDied","Data":"3fd788a750903d4c04588d71953cf2a84ab583b3c79d272d54e165a80c078510"} Oct 30 01:28:14.387617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.387597 5055 scope.go:122] "RemoveContainer" containerID="832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa" Oct 30 01:28:14.399083 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.399061 5055 scope.go:122] "RemoveContainer" containerID="8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541" Oct 30 01:28:14.407586 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.407570 5055 scope.go:122] "RemoveContainer" containerID="832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa" Oct 30 01:28:14.407840 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:28:14.407820 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa\": container with ID starting with 832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa not found: ID does not exist" containerID="832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa" Oct 30 01:28:14.407926 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.407849 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa"} err="failed to get container status \"832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa\": rpc error: code = NotFound desc = could not find container \"832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa\": container with ID starting with 832ce02766a8d0686e170cbb707770e475a857c9370d64c104f906ae1807bfaa not found: ID does not exist" Oct 30 01:28:14.407926 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.407867 5055 scope.go:122] "RemoveContainer" containerID="8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541" Oct 30 01:28:14.408128 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:28:14.408113 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541\": container with ID starting with 8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541 not found: ID does not exist" containerID="8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541" Oct 30 01:28:14.408164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.408133 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541"} err="failed to get container status \"8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541\": rpc error: code = NotFound desc = could not find container \"8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541\": container with ID starting with 8a9475fb24eca9bed4d8df7a1e1b54a83b2b940dbe5849dde8efe2c4cfcd5541 not found: ID does not exist" Oct 30 01:28:14.441273 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.441246 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v"] Oct 30 01:28:14.452995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.452973 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-18c05df8-openshift-default-f56ff45c-ssv4v"] Oct 30 01:28:14.819892 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:14.819866 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" path="/var/lib/kubelet/pods/b4fc8e9d-801f-435a-8cb5-556a8af10d93/volumes" Oct 30 01:28:17.417447 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.417396 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw"] Oct 30 01:28:17.417958 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.417946 5055 cpu_manager.go:414] "containerMap: removing container" podUID="95d0a2a8-12be-4714-a14b-4bd6c253efb7" containerName="payload-pre-processing" Oct 30 01:28:17.418020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.417961 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d0a2a8-12be-4714-a14b-4bd6c253efb7" containerName="payload-pre-processing" Oct 30 01:28:17.418020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.417972 5055 cpu_manager.go:414] "containerMap: removing container" podUID="314ef409-5c59-4abb-9d47-46a916fae95f" containerName="payload-pre-processing" Oct 30 01:28:17.418020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.417977 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314ef409-5c59-4abb-9d47-46a916fae95f" containerName="payload-pre-processing" Oct 30 01:28:17.418020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.417986 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" Oct 30 01:28:17.418020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.417991 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" Oct 30 01:28:17.418020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.418017 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" Oct 30 01:28:17.418020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.418021 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" Oct 30 01:28:17.418385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.418083 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d0a2a8-12be-4714-a14b-4bd6c253efb7" containerName="payload-pre-processing" Oct 30 01:28:17.418385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.418091 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="314ef409-5c59-4abb-9d47-46a916fae95f" containerName="payload-pre-processing" Oct 30 01:28:17.418385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.418099 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" Oct 30 01:28:17.426302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.426279 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.453755 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.453720 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"default-dockercfg-bz6bf\"" Oct 30 01:28:17.453888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.453869 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"simulator-w3-558327-kserve-self-signed-certs\"" Oct 30 01:28:17.455596 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.455572 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw"] Oct 30 01:28:17.471520 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.471497 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"kube-root-ca.crt\"" Oct 30 01:28:17.471651 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.471576 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"openshift-service-ca.crt\"" Oct 30 01:28:17.595553 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.595523 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tls-certs\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.595717 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.595564 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-home\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.595717 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.595598 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j98vz\" (UniqueName: \"kubernetes.io/projected/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-kube-api-access-j98vz\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.595844 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.595742 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tmp-dir\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.595844 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.595801 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-model-cache\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.595918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.595884 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-dshm\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.699921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.699821 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j98vz\" (UniqueName: \"kubernetes.io/projected/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-kube-api-access-j98vz\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.700173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.700142 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tmp-dir\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.700303 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.700205 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-model-cache\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.700303 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.700252 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-dshm\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.700464 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.700401 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tls-certs\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.700464 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.700446 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-home\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.701533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.700863 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-home\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.701533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.701489 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tmp-dir\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.701792 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.701768 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-model-cache\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.708051 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.706904 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tls-certs\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.709844 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.709590 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-dshm\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.722831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.722810 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j98vz\" (UniqueName: \"kubernetes.io/projected/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-kube-api-access-j98vz\") pod \"simulator-w3-558327-kserve-695db7695d-nl9qw\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:17.747649 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:17.747629 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:18.052842 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:28:18.052140 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ba9b42b_7151_450b_9287_e2a2f8bc4deb.slice/crio-588e3ce836f998c9435d4eeaa307bb81d33336c2af6b206d43106e297da078b1 WatchSource:0}: Error finding container 588e3ce836f998c9435d4eeaa307bb81d33336c2af6b206d43106e297da078b1: Status 404 returned error can't find the container with id 588e3ce836f998c9435d4eeaa307bb81d33336c2af6b206d43106e297da078b1 Oct 30 01:28:18.052842 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:18.052730 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw"] Oct 30 01:28:18.418279 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:18.418200 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" event={"ID":"9ba9b42b-7151-450b-9287-e2a2f8bc4deb","Type":"ContainerStarted","Data":"588e3ce836f998c9435d4eeaa307bb81d33336c2af6b206d43106e297da078b1"} Oct 30 01:28:19.227142 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:28:19.227104 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:28:20.434281 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:20.434253 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" event={"ID":"9ba9b42b-7151-450b-9287-e2a2f8bc4deb","Type":"ContainerStarted","Data":"a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354"} Oct 30 01:28:20.474564 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:20.474519 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podStartSLOduration=2.177346537 podStartE2EDuration="3.474505276s" totalImagesPullingTime="1.297158739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:20.473723239 +0000 UTC m=+667.794680292" watchObservedRunningTime="2026-10-30 01:28:20.474505276 +0000 UTC m=+667.795462316" Oct 30 01:28:20.748187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:20.748139 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:22.849125 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.849048 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f"] Oct 30 01:28:22.849613 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.849598 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fc8e9d-801f-435a-8cb5-556a8af10d93" containerName="istio-proxy" Oct 30 01:28:22.852732 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.852717 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:22.886688 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.886642 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-cf82f8-serving-cert\"" Oct 30 01:28:22.903990 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.903959 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 01:28:22.904281 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.904266 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x5cst\"" Oct 30 01:28:22.907652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.907625 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f"] Oct 30 01:28:22.960854 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.960825 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-metrics-certs\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:22.961004 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.960900 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-maas-api-tls\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:22.961047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.960995 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:22.961080 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:22.961043 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrj56\" (UniqueName: \"kubernetes.io/projected/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-kube-api-access-nrj56\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:23.062095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:23.061838 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:23.062095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:23.061896 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrj56\" (UniqueName: \"kubernetes.io/projected/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-kube-api-access-nrj56\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:23.062095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:23.061995 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-metrics-certs\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:23.062470 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:23.062105 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-maas-api-tls\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:23.066672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:23.066427 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:23.079351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:23.075853 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-metrics-certs\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:23.094885 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:23.094853 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-maas-api-tls\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:23.095183 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:23.095161 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrj56\" (UniqueName: \"kubernetes.io/projected/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-kube-api-access-nrj56\") pod \"maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:23.183607 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:23.183511 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:23.488495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:23.488469 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f"] Oct 30 01:28:23.491727 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:28:23.491700 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b20e1b_1132_49c6_a431_e93ccf0e7bdb.slice/crio-b34db897ab9bab9e5b732c8590681930277d506b7cb986df9c61633394516731 WatchSource:0}: Error finding container b34db897ab9bab9e5b732c8590681930277d506b7cb986df9c61633394516731: Status 404 returned error can't find the container with id b34db897ab9bab9e5b732c8590681930277d506b7cb986df9c61633394516731 Oct 30 01:28:24.459048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:24.459011 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" event={"ID":"45b20e1b-1132-49c6-a431-e93ccf0e7bdb","Type":"ContainerStarted","Data":"b34db897ab9bab9e5b732c8590681930277d506b7cb986df9c61633394516731"} Oct 30 01:28:26.495054 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:26.495019 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" event={"ID":"45b20e1b-1132-49c6-a431-e93ccf0e7bdb","Type":"ContainerStarted","Data":"e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d"} Oct 30 01:28:26.495628 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:26.495180 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:26.538416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:26.538363 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" podStartSLOduration=2.328352688 podStartE2EDuration="4.538349557s" totalImagesPullingTime="2.209996869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:26.536173424 +0000 UTC m=+673.857130465" watchObservedRunningTime="2026-10-30 01:28:26.538349557 +0000 UTC m=+673.859306596" Oct 30 01:28:27.747970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:27.747933 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:27.760954 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:27.760931 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:28.529233 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:28.529201 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:28:29.039746 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.039699 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf"] Oct 30 01:28:29.046660 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.046630 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.065663 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.065636 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf"] Oct 30 01:28:29.074826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.074788 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-eda881\"/\"openshift-service-ca.crt\"" Oct 30 01:28:29.075222 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.075204 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eda881\"/\"default-dockercfg-fg5tb\"" Oct 30 01:28:29.075485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.075471 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eda881\"/\"simulator-w2-eda881-kserve-self-signed-certs\"" Oct 30 01:28:29.075695 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.075636 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-eda881\"/\"kube-root-ca.crt\"" Oct 30 01:28:29.172525 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.172488 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-model-cache\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.172700 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.172666 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqnnj\" (UniqueName: \"kubernetes.io/projected/488e9de5-05b4-48ed-a23a-f4db797631df-kube-api-access-wqnnj\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.172749 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.172716 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-tmp-dir\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.172781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.172756 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488e9de5-05b4-48ed-a23a-f4db797631df-tls-certs\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.172814 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.172802 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-dshm\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.172870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.172854 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-home\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.274085 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.274035 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488e9de5-05b4-48ed-a23a-f4db797631df-tls-certs\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.274273 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.274110 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-dshm\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.274273 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.274157 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-home\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.274273 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.274214 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-model-cache\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.274459 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.274339 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqnnj\" (UniqueName: \"kubernetes.io/projected/488e9de5-05b4-48ed-a23a-f4db797631df-kube-api-access-wqnnj\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.274459 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.274374 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-tmp-dir\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.274785 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.274555 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-home\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.275258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.275213 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-model-cache\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.275418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.275397 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-tmp-dir\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.276901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.276872 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488e9de5-05b4-48ed-a23a-f4db797631df-tls-certs\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.277197 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.277168 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-dshm\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.295134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.295061 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqnnj\" (UniqueName: \"kubernetes.io/projected/488e9de5-05b4-48ed-a23a-f4db797631df-kube-api-access-wqnnj\") pod \"simulator-w2-eda881-kserve-6cc464684d-4cmtf\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.364165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.363911 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:29.695368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:29.693760 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf"] Oct 30 01:28:29.698546 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:28:29.698514 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod488e9de5_05b4_48ed_a23a_f4db797631df.slice/crio-75b349f848127300b1e61e3f22185971733ea57142da95dce5599fcc9fafd9f3 WatchSource:0}: Error finding container 75b349f848127300b1e61e3f22185971733ea57142da95dce5599fcc9fafd9f3: Status 404 returned error can't find the container with id 75b349f848127300b1e61e3f22185971733ea57142da95dce5599fcc9fafd9f3 Oct 30 01:28:30.527546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:30.527508 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" event={"ID":"488e9de5-05b4-48ed-a23a-f4db797631df","Type":"ContainerStarted","Data":"aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b"} Oct 30 01:28:30.527546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:30.527549 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" event={"ID":"488e9de5-05b4-48ed-a23a-f4db797631df","Type":"ContainerStarted","Data":"75b349f848127300b1e61e3f22185971733ea57142da95dce5599fcc9fafd9f3"} Oct 30 01:28:30.570721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:30.570672 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podStartSLOduration=1.570654764 podStartE2EDuration="1.570654764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:30.570086716 +0000 UTC m=+677.891043755" watchObservedRunningTime="2026-10-30 01:28:30.570654764 +0000 UTC m=+677.891611804" Oct 30 01:28:31.365267 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.365231 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:31.390742 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.390711 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s"] Oct 30 01:28:31.395260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.395235 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.425536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.425503 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"kube-root-ca.crt\"" Oct 30 01:28:31.425909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.425863 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"simulator-w0-a3a2ac-kserve-self-signed-certs\"" Oct 30 01:28:31.426143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.426125 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"openshift-service-ca.crt\"" Oct 30 01:28:31.426449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.426433 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"default-dockercfg-zpcmr\"" Oct 30 01:28:31.428821 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.428795 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s"] Oct 30 01:28:31.500120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.500074 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-dshm\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.500316 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.500125 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-tmp-dir\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.500400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.500313 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-model-cache\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.500507 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.500485 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlnv5\" (UniqueName: \"kubernetes.io/projected/4a970cad-24eb-47e8-ab37-623356dab2e4-kube-api-access-vlnv5\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.500565 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.500526 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a970cad-24eb-47e8-ab37-623356dab2e4-tls-certs\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.500619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.500564 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-home\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.601973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.601935 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-dshm\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.601973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.601970 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-tmp-dir\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.602528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.602013 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-model-cache\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.602528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.602102 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlnv5\" (UniqueName: \"kubernetes.io/projected/4a970cad-24eb-47e8-ab37-623356dab2e4-kube-api-access-vlnv5\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.602528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.602119 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a970cad-24eb-47e8-ab37-623356dab2e4-tls-certs\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.602528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.602140 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-home\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.602528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.602511 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-tmp-dir\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.602765 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.602543 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-home\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.603077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.603051 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-model-cache\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.605496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.605470 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-dshm\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.605639 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.605539 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a970cad-24eb-47e8-ab37-623356dab2e4-tls-certs\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.624238 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.624166 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlnv5\" (UniqueName: \"kubernetes.io/projected/4a970cad-24eb-47e8-ab37-623356dab2e4-kube-api-access-vlnv5\") pod \"simulator-w0-a3a2ac-kserve-776987ff8-s5v8s\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:31.725811 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:31.725779 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:32.025849 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.025824 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s"] Oct 30 01:28:32.026009 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:28:32.025975 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a970cad_24eb_47e8_ab37_623356dab2e4.slice/crio-acce4b707391aa778df6ebc7ba3a556065b690a7fc0b78a5eb861a8a4439904b WatchSource:0}: Error finding container acce4b707391aa778df6ebc7ba3a556065b690a7fc0b78a5eb861a8a4439904b: Status 404 returned error can't find the container with id acce4b707391aa778df6ebc7ba3a556065b690a7fc0b78a5eb861a8a4439904b Oct 30 01:28:32.509517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.509485 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 01:28:32.545634 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.545598 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" event={"ID":"4a970cad-24eb-47e8-ab37-623356dab2e4","Type":"ContainerStarted","Data":"46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43"} Oct 30 01:28:32.545790 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.545645 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" event={"ID":"4a970cad-24eb-47e8-ab37-623356dab2e4","Type":"ContainerStarted","Data":"acce4b707391aa778df6ebc7ba3a556065b690a7fc0b78a5eb861a8a4439904b"} Oct 30 01:28:32.547342 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.547309 5055 generic.go:356] "Generic (PLEG): container finished" podID="0d144a49-a6b5-4282-b2af-b7945045301a" containerID="28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f" exitCode=0 Oct 30 01:28:32.547486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.547359 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" event={"ID":"0d144a49-a6b5-4282-b2af-b7945045301a","Type":"ContainerDied","Data":"28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f"} Oct 30 01:28:32.547813 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.547796 5055 scope.go:122] "RemoveContainer" containerID="28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f" Oct 30 01:28:32.549670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.549282 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:28:32.549670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.549372 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:28:32.549670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.549407 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:28:32.631598 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.631542 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podStartSLOduration=1.631522921 podStartE2EDuration="1.631522921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:32.630047945 +0000 UTC m=+679.951004985" watchObservedRunningTime="2026-10-30 01:28:32.631522921 +0000 UTC m=+679.952479962" Oct 30 01:28:32.726813 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:32.726783 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:33.555491 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:33.555454 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" event={"ID":"0d144a49-a6b5-4282-b2af-b7945045301a","Type":"ContainerStarted","Data":"234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c"} Oct 30 01:28:34.312113 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:34.312083 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:34.427823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:34.427789 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:34.429077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:34.429055 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:28:34.429167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:34.429105 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" podUID="0d144a49-a6b5-4282-b2af-b7945045301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 01:28:35.428447 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:35.428411 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:28:35.428845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:35.428478 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" podUID="0d144a49-a6b5-4282-b2af-b7945045301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 01:28:36.428639 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:36.428601 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:28:36.429039 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:36.428655 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" podUID="0d144a49-a6b5-4282-b2af-b7945045301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 01:28:37.427974 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:37.427937 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:28:37.428143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:37.428018 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" podUID="0d144a49-a6b5-4282-b2af-b7945045301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 01:28:38.428009 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:38.427972 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:28:38.428411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:38.428030 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" podUID="0d144a49-a6b5-4282-b2af-b7945045301a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 01:28:39.365293 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:39.365258 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:39.378832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:39.378806 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:39.434106 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:39.434076 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:39.435667 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:39.435635 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 01:28:39.625394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:39.625302 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:28:41.726909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:41.726872 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:41.740409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:41.740387 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:42.642591 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:42.642563 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:28:49.222036 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:28:49.222001 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:28:50.600497 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.600460 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7"] Oct 30 01:28:50.603896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.603875 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.632143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.632089 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"default-dockercfg-d6pv6\"" Oct 30 01:28:50.632143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.632104 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"simulator-w1-cf82f8-kserve-self-signed-certs\"" Oct 30 01:28:50.632143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.632098 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"openshift-service-ca.crt\"" Oct 30 01:28:50.632436 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.632149 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"kube-root-ca.crt\"" Oct 30 01:28:50.634582 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.634561 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7"] Oct 30 01:28:50.741295 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.741254 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9j8n\" (UniqueName: \"kubernetes.io/projected/ee3da2e2-7a92-46c9-8bde-902d968766a4-kube-api-access-g9j8n\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.741295 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.741289 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-home\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.741580 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.741320 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-tmp-dir\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.741580 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.741425 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-model-cache\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.741580 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.741487 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-dshm\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.741580 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.741562 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3da2e2-7a92-46c9-8bde-902d968766a4-tls-certs\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.842692 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.842652 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3da2e2-7a92-46c9-8bde-902d968766a4-tls-certs\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.843008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.842750 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9j8n\" (UniqueName: \"kubernetes.io/projected/ee3da2e2-7a92-46c9-8bde-902d968766a4-kube-api-access-g9j8n\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.843008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.842778 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-home\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.843008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.842823 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-tmp-dir\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.843008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.842851 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-model-cache\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.843008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.842887 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-dshm\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.843731 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.843703 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-home\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.843731 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.843728 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-tmp-dir\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.843926 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.843826 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-model-cache\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.846445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.845893 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-dshm\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.848502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.848478 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3da2e2-7a92-46c9-8bde-902d968766a4-tls-certs\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.863750 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.863692 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9j8n\" (UniqueName: \"kubernetes.io/projected/ee3da2e2-7a92-46c9-8bde-902d968766a4-kube-api-access-g9j8n\") pod \"simulator-w1-cf82f8-kserve-565bfb8887-mb8p7\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:50.916818 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:50.916786 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:51.217096 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:51.217059 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7"] Oct 30 01:28:51.217354 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:28:51.217306 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee3da2e2_7a92_46c9_8bde_902d968766a4.slice/crio-a095a879acfabd247b2aae745432ae22be5ca5f0ae9704004d766800a79bae92 WatchSource:0}: Error finding container a095a879acfabd247b2aae745432ae22be5ca5f0ae9704004d766800a79bae92: Status 404 returned error can't find the container with id a095a879acfabd247b2aae745432ae22be5ca5f0ae9704004d766800a79bae92 Oct 30 01:28:51.690996 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:51.690913 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" event={"ID":"ee3da2e2-7a92-46c9-8bde-902d968766a4","Type":"ContainerStarted","Data":"a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91"} Oct 30 01:28:51.690996 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:51.690956 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" event={"ID":"ee3da2e2-7a92-46c9-8bde-902d968766a4","Type":"ContainerStarted","Data":"a095a879acfabd247b2aae745432ae22be5ca5f0ae9704004d766800a79bae92"} Oct 30 01:28:51.732984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:51.732928 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podStartSLOduration=1.732908917 podStartE2EDuration="1.732908917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:51.73081823 +0000 UTC m=+699.051775295" watchObservedRunningTime="2026-10-30 01:28:51.732908917 +0000 UTC m=+699.053865958" Oct 30 01:28:51.917403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:51.917366 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:28:57.747755 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.747720 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx"] Oct 30 01:28:57.758681 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.752111 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.781568 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.781544 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-eda881\"/\"premium-w2-eda881-kserve-self-signed-certs\"" Oct 30 01:28:57.783621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.783586 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx"] Oct 30 01:28:57.821501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.821471 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-home\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.821501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.821505 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tls-certs\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.821669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.821647 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvhl\" (UniqueName: \"kubernetes.io/projected/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-kube-api-access-6kvhl\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.821789 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.821767 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-dshm\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.821837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.821820 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tmp-dir\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.821878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.821864 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-model-cache\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.923064 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.923035 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvhl\" (UniqueName: \"kubernetes.io/projected/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-kube-api-access-6kvhl\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.923225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.923090 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-dshm\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.923225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.923114 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tmp-dir\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.923225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.923135 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-model-cache\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.923225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.923175 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-home\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.923225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.923203 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tls-certs\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.923609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.923585 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tmp-dir\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.923782 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.923742 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-model-cache\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.923866 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.923774 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-home\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.925654 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.925636 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tls-certs\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.926005 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.925992 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-dshm\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:57.950718 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:57.950693 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvhl\" (UniqueName: \"kubernetes.io/projected/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-kube-api-access-6kvhl\") pod \"premium-w2-eda881-kserve-845596dd65-9dmmx\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:58.065968 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:58.065877 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:28:58.370658 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:28:58.370630 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda120922c_cb9e_4f13_81aa_0b68ccdf0f9c.slice/crio-c7b1a82b3a9ea2d26be6001bf4777bbae60a2d8d45aa3d6971d37fc0c815d1f0 WatchSource:0}: Error finding container c7b1a82b3a9ea2d26be6001bf4777bbae60a2d8d45aa3d6971d37fc0c815d1f0: Status 404 returned error can't find the container with id c7b1a82b3a9ea2d26be6001bf4777bbae60a2d8d45aa3d6971d37fc0c815d1f0 Oct 30 01:28:58.371457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:58.371428 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx"] Oct 30 01:28:58.732030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:58.731998 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" event={"ID":"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c","Type":"ContainerStarted","Data":"d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7"} Oct 30 01:28:58.732244 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:58.732042 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" event={"ID":"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c","Type":"ContainerStarted","Data":"c7b1a82b3a9ea2d26be6001bf4777bbae60a2d8d45aa3d6971d37fc0c815d1f0"} Oct 30 01:28:58.776021 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:58.775965 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podStartSLOduration=1.775951088 podStartE2EDuration="1.775951088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:58.775480962 +0000 UTC m=+706.096438005" watchObservedRunningTime="2026-10-30 01:28:58.775951088 +0000 UTC m=+706.096908153" Oct 30 01:28:59.066988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:28:59.066913 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:29:00.917056 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:00.917015 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:29:00.935593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:00.931909 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:29:01.760529 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:01.760500 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:29:08.066880 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:08.066845 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:29:08.079848 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:08.079823 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:29:08.805456 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:08.805420 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:29:19.217159 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:29:19.217119 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:29:20.250560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.250519 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx"] Oct 30 01:29:20.255026 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.255000 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.285006 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.284983 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"premium-w1-cf82f8-kserve-self-signed-certs\"" Oct 30 01:29:20.287041 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.287016 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx"] Oct 30 01:29:20.356408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.356370 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5v9w\" (UniqueName: \"kubernetes.io/projected/520cd461-c87c-43a8-acea-4a669a89158e-kube-api-access-f5v9w\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.356637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.356602 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-model-cache\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.356767 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.356701 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520cd461-c87c-43a8-acea-4a669a89158e-tls-certs\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.356846 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.356822 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-dshm\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.356973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.356861 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-tmp-dir\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.356973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.356927 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-home\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.458594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.458554 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-home\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.458787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.458637 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5v9w\" (UniqueName: \"kubernetes.io/projected/520cd461-c87c-43a8-acea-4a669a89158e-kube-api-access-f5v9w\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.458787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.458668 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-model-cache\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.458787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.458759 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520cd461-c87c-43a8-acea-4a669a89158e-tls-certs\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.458955 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.458841 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-dshm\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.458955 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.458861 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-tmp-dir\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.459364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.459342 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-tmp-dir\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.459599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.459583 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-home\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.460121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.460098 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-model-cache\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.471259 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.471230 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520cd461-c87c-43a8-acea-4a669a89158e-tls-certs\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.473670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.473618 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-dshm\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.484478 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.484453 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5v9w\" (UniqueName: \"kubernetes.io/projected/520cd461-c87c-43a8-acea-4a669a89158e-kube-api-access-f5v9w\") pod \"premium-w1-cf82f8-kserve-6f99b48b66-mn4sx\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.567770 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.567705 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:20.867551 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:29:20.867523 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod520cd461_c87c_43a8_acea_4a669a89158e.slice/crio-0038dbb66644325ecf0828ca65ebd187144c46b6e4503923b69250f4f5221620 WatchSource:0}: Error finding container 0038dbb66644325ecf0828ca65ebd187144c46b6e4503923b69250f4f5221620: Status 404 returned error can't find the container with id 0038dbb66644325ecf0828ca65ebd187144c46b6e4503923b69250f4f5221620 Oct 30 01:29:20.867861 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:20.867840 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx"] Oct 30 01:29:21.876205 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:21.876165 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" event={"ID":"520cd461-c87c-43a8-acea-4a669a89158e","Type":"ContainerStarted","Data":"1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1"} Oct 30 01:29:21.876721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:21.876218 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" event={"ID":"520cd461-c87c-43a8-acea-4a669a89158e","Type":"ContainerStarted","Data":"0038dbb66644325ecf0828ca65ebd187144c46b6e4503923b69250f4f5221620"} Oct 30 01:29:21.918606 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:21.918554 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podStartSLOduration=1.9185408590000002 podStartE2EDuration="1.918540859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:21.917477477 +0000 UTC m=+729.238434514" watchObservedRunningTime="2026-10-30 01:29:21.918540859 +0000 UTC m=+729.239497899" Oct 30 01:29:22.568709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:22.568676 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:30.568909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:30.568870 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:30.584468 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:30.584441 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:30.940018 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:30.939947 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:29:49.217350 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:29:49.217292 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:29:56.823373 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:56.823263 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5"] Oct 30 01:29:56.826124 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:56.826099 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:56.843443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:56.843406 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:29:56.865245 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:56.865212 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:29:56.866215 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:56.866187 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5"] Oct 30 01:29:56.932479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:56.932443 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5012fc8d-pz5v5\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:56.932675 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:56.932621 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5012fc8d-pz5v5\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:56.932765 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:56.932749 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbtz\" (UniqueName: \"kubernetes.io/projected/7c170e56-00cc-4a35-a643-df4d790629b6-kube-api-access-vhbtz\") pod \"maas-api-revoke-sub-5012fc8d-pz5v5\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:57.034090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:57.034057 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5012fc8d-pz5v5\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:57.034299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:57.034147 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbtz\" (UniqueName: \"kubernetes.io/projected/7c170e56-00cc-4a35-a643-df4d790629b6-kube-api-access-vhbtz\") pod \"maas-api-revoke-sub-5012fc8d-pz5v5\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:57.034299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:57.034180 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5012fc8d-pz5v5\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:57.034819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:57.034798 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5012fc8d-pz5v5\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:57.036672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:57.036647 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5012fc8d-pz5v5\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:57.058179 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:57.058149 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbtz\" (UniqueName: \"kubernetes.io/projected/7c170e56-00cc-4a35-a643-df4d790629b6-kube-api-access-vhbtz\") pod \"maas-api-revoke-sub-5012fc8d-pz5v5\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:57.140057 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:57.139965 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:29:57.441958 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:29:57.441882 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c170e56_00cc_4a35_a643_df4d790629b6.slice/crio-adb1b3f683d13dcb622c702bec5605e5b0ed1fb21b50296514db8fb1b698a362 WatchSource:0}: Error finding container adb1b3f683d13dcb622c702bec5605e5b0ed1fb21b50296514db8fb1b698a362: Status 404 returned error can't find the container with id adb1b3f683d13dcb622c702bec5605e5b0ed1fb21b50296514db8fb1b698a362 Oct 30 01:29:57.442869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:57.442796 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5"] Oct 30 01:29:58.087181 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:58.087101 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" event={"ID":"7c170e56-00cc-4a35-a643-df4d790629b6","Type":"ContainerStarted","Data":"74aa770af2ea411335750841d5f1742c67fae29dcaf0d47c38224bf682bb0656"} Oct 30 01:29:58.087181 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:58.087154 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" event={"ID":"7c170e56-00cc-4a35-a643-df4d790629b6","Type":"ContainerStarted","Data":"adb1b3f683d13dcb622c702bec5605e5b0ed1fb21b50296514db8fb1b698a362"} Oct 30 01:29:58.129123 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:58.129078 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" podStartSLOduration=2.129064439 podStartE2EDuration="2.129064439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:58.128893585 +0000 UTC m=+765.449850625" watchObservedRunningTime="2026-10-30 01:29:58.129064439 +0000 UTC m=+765.450021478" Oct 30 01:29:59.095189 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:59.095152 5055 generic.go:356] "Generic (PLEG): container finished" podID="7c170e56-00cc-4a35-a643-df4d790629b6" containerID="74aa770af2ea411335750841d5f1742c67fae29dcaf0d47c38224bf682bb0656" exitCode=0 Oct 30 01:29:59.095597 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:29:59.095239 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" event={"ID":"7c170e56-00cc-4a35-a643-df4d790629b6","Type":"ContainerDied","Data":"74aa770af2ea411335750841d5f1742c67fae29dcaf0d47c38224bf682bb0656"} Oct 30 01:30:00.172817 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.172786 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw"] Oct 30 01:30:00.179021 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.178993 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" Oct 30 01:30:00.196171 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.196143 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p"] Oct 30 01:30:00.201808 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.201782 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw"] Oct 30 01:30:00.201946 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.201882 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" Oct 30 01:30:00.229679 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.229654 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p"] Oct 30 01:30:00.263752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.263732 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:30:00.270159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.270136 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw\" (UID: \"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" Oct 30 01:30:00.270335 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.270307 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggxmb\" (UniqueName: \"kubernetes.io/projected/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-kube-api-access-ggxmb\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw\" (UID: \"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" Oct 30 01:30:00.371598 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.371529 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-openshift-ca\") pod \"7c170e56-00cc-4a35-a643-df4d790629b6\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " Oct 30 01:30:00.371742 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.371610 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-xks-ca\") pod \"7c170e56-00cc-4a35-a643-df4d790629b6\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " Oct 30 01:30:00.371742 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.371666 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c170e56-00cc-4a35-a643-df4d790629b6-kube-api-access-vhbtz\" (UniqueName: \"kubernetes.io/projected/7c170e56-00cc-4a35-a643-df4d790629b6-kube-api-access-vhbtz\") pod \"7c170e56-00cc-4a35-a643-df4d790629b6\" (UID: \"7c170e56-00cc-4a35-a643-df4d790629b6\") " Oct 30 01:30:00.371867 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.371782 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56014cd-5152-47db-a89b-fcf6e71f5dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p\" (UID: \"a56014cd-5152-47db-a89b-fcf6e71f5dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" Oct 30 01:30:00.371940 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.371914 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-openshift-ca" pod "7c170e56-00cc-4a35-a643-df4d790629b6" (UID: "7c170e56-00cc-4a35-a643-df4d790629b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:00.372003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.371926 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggxmb\" (UniqueName: \"kubernetes.io/projected/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-kube-api-access-ggxmb\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw\" (UID: \"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" Oct 30 01:30:00.372051 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.372002 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pxql\" (UniqueName: \"kubernetes.io/projected/a56014cd-5152-47db-a89b-fcf6e71f5dfd-kube-api-access-4pxql\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p\" (UID: \"a56014cd-5152-47db-a89b-fcf6e71f5dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" Oct 30 01:30:00.372051 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.372042 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw\" (UID: \"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" Oct 30 01:30:00.372136 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.372121 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-openshift-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:00.372713 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.372689 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw\" (UID: \"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" Oct 30 01:30:00.373985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.373958 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-xks-ca" pod "7c170e56-00cc-4a35-a643-df4d790629b6" (UID: "7c170e56-00cc-4a35-a643-df4d790629b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:00.374073 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.374056 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c170e56-00cc-4a35-a643-df4d790629b6-kube-api-access-vhbtz" pod "7c170e56-00cc-4a35-a643-df4d790629b6" (UID: "7c170e56-00cc-4a35-a643-df4d790629b6"). InnerVolumeSpecName "kube-api-access-vhbtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:00.391679 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.391657 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggxmb\" (UniqueName: \"kubernetes.io/projected/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-kube-api-access-ggxmb\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw\" (UID: \"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" Oct 30 01:30:00.472686 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.472656 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pxql\" (UniqueName: \"kubernetes.io/projected/a56014cd-5152-47db-a89b-fcf6e71f5dfd-kube-api-access-4pxql\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p\" (UID: \"a56014cd-5152-47db-a89b-fcf6e71f5dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" Oct 30 01:30:00.472855 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.472712 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56014cd-5152-47db-a89b-fcf6e71f5dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p\" (UID: \"a56014cd-5152-47db-a89b-fcf6e71f5dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" Oct 30 01:30:00.472855 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.472758 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c170e56-00cc-4a35-a643-df4d790629b6-maas-api-xks-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:00.472855 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.472767 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhbtz\" (UniqueName: \"kubernetes.io/projected/7c170e56-00cc-4a35-a643-df4d790629b6-kube-api-access-vhbtz\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:00.473281 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.473265 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56014cd-5152-47db-a89b-fcf6e71f5dfd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p\" (UID: \"a56014cd-5152-47db-a89b-fcf6e71f5dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" Oct 30 01:30:00.492724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.492695 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pxql\" (UniqueName: \"kubernetes.io/projected/a56014cd-5152-47db-a89b-fcf6e71f5dfd-kube-api-access-4pxql\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p\" (UID: \"a56014cd-5152-47db-a89b-fcf6e71f5dfd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" Oct 30 01:30:00.502529 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.502507 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" Oct 30 01:30:00.515539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.515515 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" Oct 30 01:30:00.810106 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:30:00.810074 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebec6d3_30ed_4c6f_83a8_d8c9048794dc.slice/crio-f36ff207740897bf1dcf14db3ef5d6106fa98087f0c05e030dc34f31b068b87f WatchSource:0}: Error finding container f36ff207740897bf1dcf14db3ef5d6106fa98087f0c05e030dc34f31b068b87f: Status 404 returned error can't find the container with id f36ff207740897bf1dcf14db3ef5d6106fa98087f0c05e030dc34f31b068b87f Oct 30 01:30:00.810790 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.810761 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw"] Oct 30 01:30:00.828643 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:00.828619 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p"] Oct 30 01:30:00.829016 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:30:00.828996 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda56014cd_5152_47db_a89b_fcf6e71f5dfd.slice/crio-7ab6b11c38303515eca9339d15c208b91c4753fd9633e8cc977fc9af5b8358b1 WatchSource:0}: Error finding container 7ab6b11c38303515eca9339d15c208b91c4753fd9633e8cc977fc9af5b8358b1: Status 404 returned error can't find the container with id 7ab6b11c38303515eca9339d15c208b91c4753fd9633e8cc977fc9af5b8358b1 Oct 30 01:30:01.109308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:01.109270 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" event={"ID":"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc","Type":"ContainerStarted","Data":"2848f9ee0c8cf655d65ca5717864cb861bf8f25f74d3dad763019cb580f3e88c"} Oct 30 01:30:01.109465 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:01.109340 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" event={"ID":"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc","Type":"ContainerStarted","Data":"f36ff207740897bf1dcf14db3ef5d6106fa98087f0c05e030dc34f31b068b87f"} Oct 30 01:30:01.110621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:01.110598 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" event={"ID":"a56014cd-5152-47db-a89b-fcf6e71f5dfd","Type":"ContainerStarted","Data":"7ab6b11c38303515eca9339d15c208b91c4753fd9633e8cc977fc9af5b8358b1"} Oct 30 01:30:01.111898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:01.111868 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" Oct 30 01:30:01.111898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:01.111883 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5" event={"ID":"7c170e56-00cc-4a35-a643-df4d790629b6","Type":"ContainerDied","Data":"adb1b3f683d13dcb622c702bec5605e5b0ed1fb21b50296514db8fb1b698a362"} Oct 30 01:30:01.112028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:01.111928 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adb1b3f683d13dcb622c702bec5605e5b0ed1fb21b50296514db8fb1b698a362" Oct 30 01:30:01.149003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:01.148961 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" podStartSLOduration=1.148948069 podStartE2EDuration="1.148948069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:01.147605719 +0000 UTC m=+768.468562759" watchObservedRunningTime="2026-10-30 01:30:01.148948069 +0000 UTC m=+768.469905109" Oct 30 01:30:02.119542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:02.119504 5055 generic.go:356] "Generic (PLEG): container finished" podID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerID="2848f9ee0c8cf655d65ca5717864cb861bf8f25f74d3dad763019cb580f3e88c" exitCode=60 Oct 30 01:30:02.120103 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:02.119590 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" event={"ID":"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc","Type":"ContainerDied","Data":"2848f9ee0c8cf655d65ca5717864cb861bf8f25f74d3dad763019cb580f3e88c"} Oct 30 01:30:02.120103 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:02.119910 5055 scope.go:122] "RemoveContainer" containerID="2848f9ee0c8cf655d65ca5717864cb861bf8f25f74d3dad763019cb580f3e88c" Oct 30 01:30:02.121360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:02.121320 5055 generic.go:356] "Generic (PLEG): container finished" podID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerID="7f6fe55281abc8f5b98b12bd1659b4f7d06524e92f2fe67ac47ad9af5c4b793f" exitCode=60 Oct 30 01:30:02.121606 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:02.121440 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" event={"ID":"a56014cd-5152-47db-a89b-fcf6e71f5dfd","Type":"ContainerDied","Data":"7f6fe55281abc8f5b98b12bd1659b4f7d06524e92f2fe67ac47ad9af5c4b793f"} Oct 30 01:30:02.121660 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:02.121616 5055 scope.go:122] "RemoveContainer" containerID="7f6fe55281abc8f5b98b12bd1659b4f7d06524e92f2fe67ac47ad9af5c4b793f" Oct 30 01:30:03.130096 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:03.130050 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" event={"ID":"a56014cd-5152-47db-a89b-fcf6e71f5dfd","Type":"ContainerStarted","Data":"6f20cce167b3bdb925c73744402658ea0dc277ba91b13cbf70ed8d5fa262d65d"} Oct 30 01:30:03.132110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:03.132086 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" event={"ID":"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc","Type":"ContainerStarted","Data":"cef0fa64d09518854d1ea751f3d1ee7a851ea8fdfab28e777500490267bfe4c3"} Oct 30 01:30:03.188460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:03.188400 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" podStartSLOduration=3.188378999 podStartE2EDuration="3.188378999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:03.183305486 +0000 UTC m=+770.504262527" watchObservedRunningTime="2026-10-30 01:30:03.188378999 +0000 UTC m=+770.509336041" Oct 30 01:30:04.139641 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:04.139605 5055 generic.go:356] "Generic (PLEG): container finished" podID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerID="cef0fa64d09518854d1ea751f3d1ee7a851ea8fdfab28e777500490267bfe4c3" exitCode=60 Oct 30 01:30:04.140100 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:04.139656 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" event={"ID":"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc","Type":"ContainerDied","Data":"cef0fa64d09518854d1ea751f3d1ee7a851ea8fdfab28e777500490267bfe4c3"} Oct 30 01:30:04.140100 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:04.139726 5055 scope.go:122] "RemoveContainer" containerID="2848f9ee0c8cf655d65ca5717864cb861bf8f25f74d3dad763019cb580f3e88c" Oct 30 01:30:04.140100 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:04.140001 5055 scope.go:122] "RemoveContainer" containerID="cef0fa64d09518854d1ea751f3d1ee7a851ea8fdfab28e777500490267bfe4c3" Oct 30 01:30:04.140306 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:30:04.140287 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw_odh-ai-gateway-infra(9ebec6d3-30ed-4c6f-83a8-d8c9048794dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" Oct 30 01:30:04.141702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:04.141679 5055 generic.go:356] "Generic (PLEG): container finished" podID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerID="6f20cce167b3bdb925c73744402658ea0dc277ba91b13cbf70ed8d5fa262d65d" exitCode=60 Oct 30 01:30:04.141797 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:04.141742 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" event={"ID":"a56014cd-5152-47db-a89b-fcf6e71f5dfd","Type":"ContainerDied","Data":"6f20cce167b3bdb925c73744402658ea0dc277ba91b13cbf70ed8d5fa262d65d"} Oct 30 01:30:04.141987 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:04.141973 5055 scope.go:122] "RemoveContainer" containerID="6f20cce167b3bdb925c73744402658ea0dc277ba91b13cbf70ed8d5fa262d65d" Oct 30 01:30:04.142213 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:30:04.142197 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p_odh-ai-gateway-infra(a56014cd-5152-47db-a89b-fcf6e71f5dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" Oct 30 01:30:04.152654 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:04.152622 5055 scope.go:122] "RemoveContainer" containerID="7f6fe55281abc8f5b98b12bd1659b4f7d06524e92f2fe67ac47ad9af5c4b793f" Oct 30 01:30:05.148832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:05.148800 5055 scope.go:122] "RemoveContainer" containerID="cef0fa64d09518854d1ea751f3d1ee7a851ea8fdfab28e777500490267bfe4c3" Oct 30 01:30:05.149293 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:30:05.149086 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw_odh-ai-gateway-infra(9ebec6d3-30ed-4c6f-83a8-d8c9048794dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" Oct 30 01:30:05.150413 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:05.150389 5055 scope.go:122] "RemoveContainer" containerID="6f20cce167b3bdb925c73744402658ea0dc277ba91b13cbf70ed8d5fa262d65d" Oct 30 01:30:05.150602 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:30:05.150584 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p_odh-ai-gateway-infra(a56014cd-5152-47db-a89b-fcf6e71f5dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" Oct 30 01:30:08.480899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.480867 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb"] Oct 30 01:30:08.481604 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.481584 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7c170e56-00cc-4a35-a643-df4d790629b6" containerName="revoke-keys" Oct 30 01:30:08.481674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.481606 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c170e56-00cc-4a35-a643-df4d790629b6" containerName="revoke-keys" Oct 30 01:30:08.481803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.481791 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c170e56-00cc-4a35-a643-df4d790629b6" containerName="revoke-keys" Oct 30 01:30:08.487216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.487189 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:08.507634 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.507600 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb"] Oct 30 01:30:08.518707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.517046 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:08.556254 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.556228 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d5c792-4d2xb\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:08.556415 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.556292 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j99sj\" (UniqueName: \"kubernetes.io/projected/27218363-120b-4c29-9656-96e9a2d96d89-kube-api-access-j99sj\") pod \"maas-api-revoke-sub-74d5c792-4d2xb\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:08.556460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.556418 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d5c792-4d2xb\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:08.657372 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.657338 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d5c792-4d2xb\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:08.657542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.657406 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99sj\" (UniqueName: \"kubernetes.io/projected/27218363-120b-4c29-9656-96e9a2d96d89-kube-api-access-j99sj\") pod \"maas-api-revoke-sub-74d5c792-4d2xb\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:08.657542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.657453 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d5c792-4d2xb\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:08.658160 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.658073 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d5c792-4d2xb\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:08.659987 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.659964 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d5c792-4d2xb\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:08.678188 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.678160 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99sj\" (UniqueName: \"kubernetes.io/projected/27218363-120b-4c29-9656-96e9a2d96d89-kube-api-access-j99sj\") pod \"maas-api-revoke-sub-74d5c792-4d2xb\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:08.802447 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:08.802373 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:09.112012 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:09.111979 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb"] Oct 30 01:30:09.113037 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:30:09.113005 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27218363_120b_4c29_9656_96e9a2d96d89.slice/crio-1bb8426a60d326d0e66d4a40ae120efa3356ff5aeef1fbf5b35d9648ca29fd1c WatchSource:0}: Error finding container 1bb8426a60d326d0e66d4a40ae120efa3356ff5aeef1fbf5b35d9648ca29fd1c: Status 404 returned error can't find the container with id 1bb8426a60d326d0e66d4a40ae120efa3356ff5aeef1fbf5b35d9648ca29fd1c Oct 30 01:30:09.173929 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:09.173899 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" event={"ID":"27218363-120b-4c29-9656-96e9a2d96d89","Type":"ContainerStarted","Data":"1bb8426a60d326d0e66d4a40ae120efa3356ff5aeef1fbf5b35d9648ca29fd1c"} Oct 30 01:30:10.181318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:10.181277 5055 generic.go:356] "Generic (PLEG): container finished" podID="27218363-120b-4c29-9656-96e9a2d96d89" containerID="a21ae8acd82c5543efb36855c9569e427731d2f1774846d9536a6af0f7430975" exitCode=0 Oct 30 01:30:10.181810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:10.181383 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" event={"ID":"27218363-120b-4c29-9656-96e9a2d96d89","Type":"ContainerDied","Data":"a21ae8acd82c5543efb36855c9569e427731d2f1774846d9536a6af0f7430975"} Oct 30 01:30:11.349623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:11.349595 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:11.386095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:11.386065 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-openshift-ca\") pod \"27218363-120b-4c29-9656-96e9a2d96d89\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " Oct 30 01:30:11.386255 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:11.386115 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-xks-ca\") pod \"27218363-120b-4c29-9656-96e9a2d96d89\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " Oct 30 01:30:11.386255 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:11.386218 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27218363-120b-4c29-9656-96e9a2d96d89-kube-api-access-j99sj\" (UniqueName: \"kubernetes.io/projected/27218363-120b-4c29-9656-96e9a2d96d89-kube-api-access-j99sj\") pod \"27218363-120b-4c29-9656-96e9a2d96d89\" (UID: \"27218363-120b-4c29-9656-96e9a2d96d89\") " Oct 30 01:30:11.386527 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:11.386500 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-openshift-ca" pod "27218363-120b-4c29-9656-96e9a2d96d89" (UID: "27218363-120b-4c29-9656-96e9a2d96d89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:11.386707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:11.386690 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-openshift-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.388456 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:11.388432 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-xks-ca" pod "27218363-120b-4c29-9656-96e9a2d96d89" (UID: "27218363-120b-4c29-9656-96e9a2d96d89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:11.388758 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:11.388741 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27218363-120b-4c29-9656-96e9a2d96d89-kube-api-access-j99sj" pod "27218363-120b-4c29-9656-96e9a2d96d89" (UID: "27218363-120b-4c29-9656-96e9a2d96d89"). InnerVolumeSpecName "kube-api-access-j99sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:11.487546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:11.487513 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27218363-120b-4c29-9656-96e9a2d96d89-maas-api-xks-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:11.487546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:11.487542 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j99sj\" (UniqueName: \"kubernetes.io/projected/27218363-120b-4c29-9656-96e9a2d96d89-kube-api-access-j99sj\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:12.197140 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:12.197104 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" event={"ID":"27218363-120b-4c29-9656-96e9a2d96d89","Type":"ContainerDied","Data":"1bb8426a60d326d0e66d4a40ae120efa3356ff5aeef1fbf5b35d9648ca29fd1c"} Oct 30 01:30:12.197370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:12.197161 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bb8426a60d326d0e66d4a40ae120efa3356ff5aeef1fbf5b35d9648ca29fd1c" Oct 30 01:30:12.197370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:12.197120 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb" Oct 30 01:30:13.813223 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:13.813193 5055 scope.go:122] "RemoveContainer" containerID="cef0fa64d09518854d1ea751f3d1ee7a851ea8fdfab28e777500490267bfe4c3" Oct 30 01:30:14.211895 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.211861 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" event={"ID":"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc","Type":"ContainerStarted","Data":"61d9ca1a10c965818a9995df257d7462cdeb5621ba0c09b798b6c6329138ae5d"} Oct 30 01:30:14.833457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.833409 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5"] Oct 30 01:30:14.834170 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.834148 5055 cpu_manager.go:414] "containerMap: removing container" podUID="27218363-120b-4c29-9656-96e9a2d96d89" containerName="revoke-keys" Oct 30 01:30:14.834170 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.834169 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27218363-120b-4c29-9656-96e9a2d96d89" containerName="revoke-keys" Oct 30 01:30:14.834317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.834301 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="27218363-120b-4c29-9656-96e9a2d96d89" containerName="revoke-keys" Oct 30 01:30:14.838046 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.838024 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:14.864190 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.864147 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5"] Oct 30 01:30:14.871764 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.871739 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:30:14.924823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.924737 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11891f4f-2nrx5\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:14.924823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.924795 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11891f4f-2nrx5\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:14.925047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:14.924918 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zbkk\" (UniqueName: \"kubernetes.io/projected/f86095a6-b32e-49b2-98c5-3a2b5925d161-kube-api-access-5zbkk\") pod \"maas-api-revoke-sub-11891f4f-2nrx5\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:15.026396 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.026355 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zbkk\" (UniqueName: \"kubernetes.io/projected/f86095a6-b32e-49b2-98c5-3a2b5925d161-kube-api-access-5zbkk\") pod \"maas-api-revoke-sub-11891f4f-2nrx5\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:15.026580 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.026442 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11891f4f-2nrx5\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:15.026580 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.026484 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11891f4f-2nrx5\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:15.027108 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.027076 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11891f4f-2nrx5\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:15.029007 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.028986 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11891f4f-2nrx5\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:15.046561 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.046530 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zbkk\" (UniqueName: \"kubernetes.io/projected/f86095a6-b32e-49b2-98c5-3a2b5925d161-kube-api-access-5zbkk\") pod \"maas-api-revoke-sub-11891f4f-2nrx5\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:15.151066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.151028 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:15.226335 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.226292 5055 generic.go:356] "Generic (PLEG): container finished" podID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerID="61d9ca1a10c965818a9995df257d7462cdeb5621ba0c09b798b6c6329138ae5d" exitCode=60 Oct 30 01:30:15.226492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.226366 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" event={"ID":"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc","Type":"ContainerDied","Data":"61d9ca1a10c965818a9995df257d7462cdeb5621ba0c09b798b6c6329138ae5d"} Oct 30 01:30:15.226492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.226448 5055 scope.go:122] "RemoveContainer" containerID="cef0fa64d09518854d1ea751f3d1ee7a851ea8fdfab28e777500490267bfe4c3" Oct 30 01:30:15.226839 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.226820 5055 scope.go:122] "RemoveContainer" containerID="61d9ca1a10c965818a9995df257d7462cdeb5621ba0c09b798b6c6329138ae5d" Oct 30 01:30:15.227850 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:30:15.227384 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw_odh-ai-gateway-infra(9ebec6d3-30ed-4c6f-83a8-d8c9048794dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" Oct 30 01:30:15.289553 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.289521 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw"] Oct 30 01:30:15.459626 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:30:15.459596 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf86095a6_b32e_49b2_98c5_3a2b5925d161.slice/crio-09e51ce7bb84aae9e44c73502bc636eeed996e68fa8fd37b6848c4ec85d6e695 WatchSource:0}: Error finding container 09e51ce7bb84aae9e44c73502bc636eeed996e68fa8fd37b6848c4ec85d6e695: Status 404 returned error can't find the container with id 09e51ce7bb84aae9e44c73502bc636eeed996e68fa8fd37b6848c4ec85d6e695 Oct 30 01:30:15.460047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:15.460027 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5"] Oct 30 01:30:16.235316 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.235285 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" event={"ID":"f86095a6-b32e-49b2-98c5-3a2b5925d161","Type":"ContainerStarted","Data":"e7dd8b1b2466638a7718c5ac231a50b4ac1f769b37fa254a2f2719fc3194c856"} Oct 30 01:30:16.235737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.235351 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" event={"ID":"f86095a6-b32e-49b2-98c5-3a2b5925d161","Type":"ContainerStarted","Data":"09e51ce7bb84aae9e44c73502bc636eeed996e68fa8fd37b6848c4ec85d6e695"} Oct 30 01:30:16.317214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.317160 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" podStartSLOduration=2.317145377 podStartE2EDuration="2.317145377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:16.314005653 +0000 UTC m=+783.634962691" watchObservedRunningTime="2026-10-30 01:30:16.317145377 +0000 UTC m=+783.638102416" Oct 30 01:30:16.470784 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.470757 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" Oct 30 01:30:16.544157 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.544062 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-kube-api-access-ggxmb\" (UniqueName: \"kubernetes.io/projected/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-kube-api-access-ggxmb\") pod \"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc\" (UID: \"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc\") " Oct 30 01:30:16.544339 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.544249 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-maas-api-service-ca\") pod \"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc\" (UID: \"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc\") " Oct 30 01:30:16.544687 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.544654 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-maas-api-service-ca" pod "9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" (UID: "9ebec6d3-30ed-4c6f-83a8-d8c9048794dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:16.546410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.546386 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-kube-api-access-ggxmb" pod "9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" (UID: "9ebec6d3-30ed-4c6f-83a8-d8c9048794dc"). InnerVolumeSpecName "kube-api-access-ggxmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:16.646364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.646304 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-maas-api-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:16.646364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.646368 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggxmb\" (UniqueName: \"kubernetes.io/projected/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc-kube-api-access-ggxmb\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:16.813112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:16.813035 5055 scope.go:122] "RemoveContainer" containerID="6f20cce167b3bdb925c73744402658ea0dc277ba91b13cbf70ed8d5fa262d65d" Oct 30 01:30:17.242730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:17.242703 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" Oct 30 01:30:17.243188 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:17.242701 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw" event={"ID":"9ebec6d3-30ed-4c6f-83a8-d8c9048794dc","Type":"ContainerDied","Data":"f36ff207740897bf1dcf14db3ef5d6106fa98087f0c05e030dc34f31b068b87f"} Oct 30 01:30:17.243188 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:17.242844 5055 scope.go:122] "RemoveContainer" containerID="61d9ca1a10c965818a9995df257d7462cdeb5621ba0c09b798b6c6329138ae5d" Oct 30 01:30:17.245209 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:17.245173 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" event={"ID":"a56014cd-5152-47db-a89b-fcf6e71f5dfd","Type":"ContainerStarted","Data":"bfc9ebcaa1d9f1b2787f0dc21469671ec4bcb110ba37c1d26d4243552e01fb82"} Oct 30 01:30:17.247533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:17.247508 5055 generic.go:356] "Generic (PLEG): container finished" podID="f86095a6-b32e-49b2-98c5-3a2b5925d161" containerID="e7dd8b1b2466638a7718c5ac231a50b4ac1f769b37fa254a2f2719fc3194c856" exitCode=0 Oct 30 01:30:17.247645 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:17.247591 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" event={"ID":"f86095a6-b32e-49b2-98c5-3a2b5925d161","Type":"ContainerDied","Data":"e7dd8b1b2466638a7718c5ac231a50b4ac1f769b37fa254a2f2719fc3194c856"} Oct 30 01:30:17.320120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:17.320082 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw"] Oct 30 01:30:17.343813 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:17.343742 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888730-np2zw"] Oct 30 01:30:18.257075 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.257042 5055 generic.go:356] "Generic (PLEG): container finished" podID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerID="bfc9ebcaa1d9f1b2787f0dc21469671ec4bcb110ba37c1d26d4243552e01fb82" exitCode=60 Oct 30 01:30:18.257599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.257120 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" event={"ID":"a56014cd-5152-47db-a89b-fcf6e71f5dfd","Type":"ContainerDied","Data":"bfc9ebcaa1d9f1b2787f0dc21469671ec4bcb110ba37c1d26d4243552e01fb82"} Oct 30 01:30:18.257599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.257191 5055 scope.go:122] "RemoveContainer" containerID="6f20cce167b3bdb925c73744402658ea0dc277ba91b13cbf70ed8d5fa262d65d" Oct 30 01:30:18.257698 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.257671 5055 scope.go:122] "RemoveContainer" containerID="bfc9ebcaa1d9f1b2787f0dc21469671ec4bcb110ba37c1d26d4243552e01fb82" Oct 30 01:30:18.258076 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:30:18.257942 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p_odh-ai-gateway-infra(a56014cd-5152-47db-a89b-fcf6e71f5dfd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" Oct 30 01:30:18.412620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.412585 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p"] Oct 30 01:30:18.421641 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.421617 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:18.466784 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.466752 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-xks-ca\") pod \"f86095a6-b32e-49b2-98c5-3a2b5925d161\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " Oct 30 01:30:18.466959 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.466943 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-openshift-ca\") pod \"f86095a6-b32e-49b2-98c5-3a2b5925d161\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " Oct 30 01:30:18.467031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.466995 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f86095a6-b32e-49b2-98c5-3a2b5925d161-kube-api-access-5zbkk\" (UniqueName: \"kubernetes.io/projected/f86095a6-b32e-49b2-98c5-3a2b5925d161-kube-api-access-5zbkk\") pod \"f86095a6-b32e-49b2-98c5-3a2b5925d161\" (UID: \"f86095a6-b32e-49b2-98c5-3a2b5925d161\") " Oct 30 01:30:18.467395 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.467363 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-openshift-ca" pod "f86095a6-b32e-49b2-98c5-3a2b5925d161" (UID: "f86095a6-b32e-49b2-98c5-3a2b5925d161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:18.469160 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.469124 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-xks-ca" pod "f86095a6-b32e-49b2-98c5-3a2b5925d161" (UID: "f86095a6-b32e-49b2-98c5-3a2b5925d161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:30:18.469262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.469244 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f86095a6-b32e-49b2-98c5-3a2b5925d161-kube-api-access-5zbkk" pod "f86095a6-b32e-49b2-98c5-3a2b5925d161" (UID: "f86095a6-b32e-49b2-98c5-3a2b5925d161"). InnerVolumeSpecName "kube-api-access-5zbkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:18.568432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.568317 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-openshift-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.568432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.568373 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zbkk\" (UniqueName: \"kubernetes.io/projected/f86095a6-b32e-49b2-98c5-3a2b5925d161-kube-api-access-5zbkk\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.568432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.568388 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f86095a6-b32e-49b2-98c5-3a2b5925d161-maas-api-xks-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:18.820811 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:18.820720 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" path="/var/lib/kubelet/pods/9ebec6d3-30ed-4c6f-83a8-d8c9048794dc/volumes" Oct 30 01:30:19.219677 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:30:19.219641 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:30:19.264916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:19.264886 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" Oct 30 01:30:19.265458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:19.264926 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5" event={"ID":"f86095a6-b32e-49b2-98c5-3a2b5925d161","Type":"ContainerDied","Data":"09e51ce7bb84aae9e44c73502bc636eeed996e68fa8fd37b6848c4ec85d6e695"} Oct 30 01:30:19.265458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:19.264976 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09e51ce7bb84aae9e44c73502bc636eeed996e68fa8fd37b6848c4ec85d6e695" Oct 30 01:30:19.425621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:19.425593 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" Oct 30 01:30:19.482638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:19.482555 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a56014cd-5152-47db-a89b-fcf6e71f5dfd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56014cd-5152-47db-a89b-fcf6e71f5dfd-maas-api-service-ca\") pod \"a56014cd-5152-47db-a89b-fcf6e71f5dfd\" (UID: \"a56014cd-5152-47db-a89b-fcf6e71f5dfd\") " Oct 30 01:30:19.482809 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:19.482674 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a56014cd-5152-47db-a89b-fcf6e71f5dfd-kube-api-access-4pxql\" (UniqueName: \"kubernetes.io/projected/a56014cd-5152-47db-a89b-fcf6e71f5dfd-kube-api-access-4pxql\") pod \"a56014cd-5152-47db-a89b-fcf6e71f5dfd\" (UID: \"a56014cd-5152-47db-a89b-fcf6e71f5dfd\") " Oct 30 01:30:19.483110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:19.483076 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a56014cd-5152-47db-a89b-fcf6e71f5dfd-maas-api-service-ca" pod "a56014cd-5152-47db-a89b-fcf6e71f5dfd" (UID: "a56014cd-5152-47db-a89b-fcf6e71f5dfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:30:19.483466 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:19.483443 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a56014cd-5152-47db-a89b-fcf6e71f5dfd-maas-api-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:19.485485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:19.485458 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56014cd-5152-47db-a89b-fcf6e71f5dfd-kube-api-access-4pxql" pod "a56014cd-5152-47db-a89b-fcf6e71f5dfd" (UID: "a56014cd-5152-47db-a89b-fcf6e71f5dfd"). InnerVolumeSpecName "kube-api-access-4pxql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:30:19.584492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:19.584455 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pxql\" (UniqueName: \"kubernetes.io/projected/a56014cd-5152-47db-a89b-fcf6e71f5dfd-kube-api-access-4pxql\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:30:20.275537 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:20.275498 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" event={"ID":"a56014cd-5152-47db-a89b-fcf6e71f5dfd","Type":"ContainerDied","Data":"7ab6b11c38303515eca9339d15c208b91c4753fd9633e8cc977fc9af5b8358b1"} Oct 30 01:30:20.276024 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:20.275571 5055 scope.go:122] "RemoveContainer" containerID="bfc9ebcaa1d9f1b2787f0dc21469671ec4bcb110ba37c1d26d4243552e01fb82" Oct 30 01:30:20.276024 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:20.275513 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p" Oct 30 01:30:20.329583 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:20.329546 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p"] Oct 30 01:30:20.342127 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:20.342079 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888730-74c5p"] Oct 30 01:30:20.819449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:20.819412 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" path="/var/lib/kubelet/pods/a56014cd-5152-47db-a89b-fcf6e71f5dfd/volumes" Oct 30 01:30:24.374848 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.374811 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9"] Oct 30 01:30:24.375357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375307 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerName="cleanup" Oct 30 01:30:24.375421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375358 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerName="cleanup" Oct 30 01:30:24.375421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375374 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerName="cleanup" Oct 30 01:30:24.375421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375380 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerName="cleanup" Oct 30 01:30:24.375421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375397 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerName="cleanup" Oct 30 01:30:24.375421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375402 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerName="cleanup" Oct 30 01:30:24.375421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375407 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f86095a6-b32e-49b2-98c5-3a2b5925d161" containerName="revoke-keys" Oct 30 01:30:24.375421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375412 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f86095a6-b32e-49b2-98c5-3a2b5925d161" containerName="revoke-keys" Oct 30 01:30:24.375630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375429 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerName="cleanup" Oct 30 01:30:24.375630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375434 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerName="cleanup" Oct 30 01:30:24.375630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375443 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerName="cleanup" Oct 30 01:30:24.375630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375447 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerName="cleanup" Oct 30 01:30:24.375630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375538 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerName="cleanup" Oct 30 01:30:24.375630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375545 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f86095a6-b32e-49b2-98c5-3a2b5925d161" containerName="revoke-keys" Oct 30 01:30:24.375630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375554 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerName="cleanup" Oct 30 01:30:24.375630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375560 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerName="cleanup" Oct 30 01:30:24.375630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.375568 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerName="cleanup" Oct 30 01:30:24.380346 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.380303 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.396188 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.396162 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-803472-openshift-default-dockercfg-hqdb6\"" Oct 30 01:30:24.412135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.412100 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9"] Oct 30 01:30:24.540059 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.540020 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.540233 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.540124 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.540233 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.540179 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.540233 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.540198 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.540384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.540277 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.540384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.540310 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.540384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.540369 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.540489 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.540442 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7brn\" (UniqueName: \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-kube-api-access-w7brn\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.540489 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.540474 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.641503 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.641395 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.641503 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.641480 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.641737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.641522 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.641737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.641560 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7brn\" (UniqueName: \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-kube-api-access-w7brn\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.641737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.641688 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.641903 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.641757 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.641903 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.641801 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.641903 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.641827 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.641903 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.641861 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.642097 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.641906 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.642097 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.642046 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.642202 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.642179 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.642439 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.642416 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.642869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.642840 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.644700 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.644673 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.644815 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.644756 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.666545 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.666516 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7brn\" (UniqueName: \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-kube-api-access-w7brn\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.667673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.667644 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:24.694864 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:24.694827 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:25.020846 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:30:25.020818 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d4eff34_fdc7_40a5_b75d_9b8028822b94.slice/crio-6ca88a5dd1eb23f051041dcde290b55530a353daf5d513543737a8042e3658ed WatchSource:0}: Error finding container 6ca88a5dd1eb23f051041dcde290b55530a353daf5d513543737a8042e3658ed: Status 404 returned error can't find the container with id 6ca88a5dd1eb23f051041dcde290b55530a353daf5d513543737a8042e3658ed Oct 30 01:30:25.023368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:25.023321 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:30:25.023448 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:25.023410 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:30:25.023448 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:25.023436 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:30:25.023854 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:25.023825 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9"] Oct 30 01:30:25.311943 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:25.311843 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" event={"ID":"8d4eff34-fdc7-40a5-b75d-9b8028822b94","Type":"ContainerStarted","Data":"335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9"} Oct 30 01:30:25.311943 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:25.311896 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" event={"ID":"8d4eff34-fdc7-40a5-b75d-9b8028822b94","Type":"ContainerStarted","Data":"6ca88a5dd1eb23f051041dcde290b55530a353daf5d513543737a8042e3658ed"} Oct 30 01:30:25.370444 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:25.370379 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" podStartSLOduration=1.370358773 podStartE2EDuration="1.370358773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:25.367285307 +0000 UTC m=+792.688242348" watchObservedRunningTime="2026-10-30 01:30:25.370358773 +0000 UTC m=+792.691315814" Oct 30 01:30:25.695966 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:25.695874 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:26.697666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:26.697632 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:30:26.698049 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:26.697699 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" podUID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 01:30:27.695617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:27.695582 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:30:27.695787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:27.695650 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" podUID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 01:30:28.696189 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:28.696154 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:30:28.696612 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:28.696212 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" podUID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 01:30:29.697353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:29.696741 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:30:29.697353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:29.696818 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" podUID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 01:30:30.695543 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:30.695504 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:30:30.695785 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:30.695565 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" podUID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Oct 30 01:30:31.867651 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:31.867618 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:31.868095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:31.868006 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:31.869414 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:31.869376 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:30:40.334939 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.334905 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk"] Oct 30 01:30:40.335458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.335424 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerName="cleanup" Oct 30 01:30:40.335458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.335437 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerName="cleanup" Oct 30 01:30:40.335558 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.335547 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ebec6d3-30ed-4c6f-83a8-d8c9048794dc" containerName="cleanup" Oct 30 01:30:40.335595 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.335559 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a56014cd-5152-47db-a89b-fcf6e71f5dfd" containerName="cleanup" Oct 30 01:30:40.339108 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.339071 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.361984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.361962 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cf82f8-openshift-default-dockercfg-xt5zt\"" Oct 30 01:30:40.376443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.376417 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk"] Oct 30 01:30:40.440461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.440429 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.440461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.440465 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.440721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.440483 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.440721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.440562 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lxcf\" (UniqueName: \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-kube-api-access-4lxcf\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.440721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.440618 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.440721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.440648 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.440935 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.440739 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.440935 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.440754 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.440935 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.440770 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.542304 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.542276 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.542508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.542310 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.542508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.542352 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.542508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.542397 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.542508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.542422 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.542508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.542446 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.542508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.542487 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lxcf\" (UniqueName: \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-kube-api-access-4lxcf\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.542809 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.542530 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.542809 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.542578 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.542922 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.542845 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.543058 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.543032 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.543187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.543091 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.543302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.543278 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.543603 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.543580 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.545286 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.545257 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.545624 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.545602 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.567214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.567176 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lxcf\" (UniqueName: \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-kube-api-access-4lxcf\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.569139 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.569118 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:40.652131 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:40.652048 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:41.020835 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:30:41.020789 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d3ec3a7_d8b6_439b_8f10_59222347d5e2.slice/crio-60bc638dfd46f063456f823718762701edccba0f0926779d09930ce335f4d555 WatchSource:0}: Error finding container 60bc638dfd46f063456f823718762701edccba0f0926779d09930ce335f4d555: Status 404 returned error can't find the container with id 60bc638dfd46f063456f823718762701edccba0f0926779d09930ce335f4d555 Oct 30 01:30:41.021214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:41.021177 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk"] Oct 30 01:30:41.023640 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:41.023611 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:30:41.023763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:41.023677 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:30:41.023763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:41.023712 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:30:41.419240 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:41.419153 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" event={"ID":"6d3ec3a7-d8b6-439b-8f10-59222347d5e2","Type":"ContainerStarted","Data":"b03e45fb17840c69433ff7b293e259a9947935202de696e3b50d07021720f845"} Oct 30 01:30:41.419240 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:41.419223 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" event={"ID":"6d3ec3a7-d8b6-439b-8f10-59222347d5e2","Type":"ContainerStarted","Data":"60bc638dfd46f063456f823718762701edccba0f0926779d09930ce335f4d555"} Oct 30 01:30:41.496020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:41.495969 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" podStartSLOduration=1.495953644 podStartE2EDuration="1.495953644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:41.482665291 +0000 UTC m=+808.803622332" watchObservedRunningTime="2026-10-30 01:30:41.495953644 +0000 UTC m=+808.816910684" Oct 30 01:30:41.652962 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:41.652926 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:42.654264 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:42.654233 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:30:42.654650 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:42.654283 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 01:30:43.652441 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:43.652405 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:30:43.652626 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:43.652467 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 01:30:44.652878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:44.652844 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:30:44.653403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:44.652904 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 01:30:45.653157 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:45.653115 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:30:45.653588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:45.653173 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 01:30:46.653414 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:46.653379 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:30:46.653917 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:46.653445 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 01:30:47.653206 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:47.653178 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:30:47.653411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:47.653228 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 01:30:48.657150 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:48.657122 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:48.657576 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:48.657541 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:48.658409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:48.658388 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:30:49.219207 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:30:49.219162 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:30:55.133150 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.133107 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps"] Oct 30 01:30:55.139143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.139112 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.174542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.174508 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-803472\"/\"kube-root-ca.crt\"" Oct 30 01:30:55.174733 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.174646 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-803472\"/\"default-dockercfg-q52mv\"" Oct 30 01:30:55.178950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.178918 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps"] Oct 30 01:30:55.179560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.179539 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-803472\"/\"distinct-w4-803472-kserve-self-signed-certs\"" Oct 30 01:30:55.179828 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.179793 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-803472\"/\"openshift-service-ca.crt\"" Oct 30 01:30:55.235423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.235384 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-model-cache\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.235615 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.235470 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-dshm\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.235615 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.235509 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-home\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.235615 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.235554 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqsqm\" (UniqueName: \"kubernetes.io/projected/d7888167-f48d-41cb-8a4a-dd9510bb9481-kube-api-access-gqsqm\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.235615 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.235616 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7888167-f48d-41cb-8a4a-dd9510bb9481-tls-certs\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.235833 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.235650 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-tmp-dir\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.337162 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.337123 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-dshm\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.337386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.337181 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-home\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.337386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.337237 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqsqm\" (UniqueName: \"kubernetes.io/projected/d7888167-f48d-41cb-8a4a-dd9510bb9481-kube-api-access-gqsqm\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.337386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.337319 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7888167-f48d-41cb-8a4a-dd9510bb9481-tls-certs\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.337566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.337388 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-tmp-dir\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.337566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.337483 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-model-cache\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.337898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.337874 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-model-cache\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.338278 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.338253 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-home\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.338589 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.338565 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-tmp-dir\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.341515 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.341488 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7888167-f48d-41cb-8a4a-dd9510bb9481-tls-certs\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.343914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.343866 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-dshm\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.373878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.368690 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqsqm\" (UniqueName: \"kubernetes.io/projected/d7888167-f48d-41cb-8a4a-dd9510bb9481-kube-api-access-gqsqm\") pod \"distinct-w4-803472-kserve-8477c5f97f-cpsps\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.456898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.456863 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:30:55.788181 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:55.788149 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps"] Oct 30 01:30:56.521403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:56.521359 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" event={"ID":"d7888167-f48d-41cb-8a4a-dd9510bb9481","Type":"ContainerStarted","Data":"5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6"} Oct 30 01:30:56.521403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:56.521416 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" event={"ID":"d7888167-f48d-41cb-8a4a-dd9510bb9481","Type":"ContainerStarted","Data":"434099926a47cd9fc2b889e960a9410516e0055893c91a942859db0605b5cec5"} Oct 30 01:30:56.571018 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:56.570963 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podStartSLOduration=1.570947888 podStartE2EDuration="1.570947888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:56.568630332 +0000 UTC m=+823.889587374" watchObservedRunningTime="2026-10-30 01:30:56.570947888 +0000 UTC m=+823.891905240" Oct 30 01:30:57.457509 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:30:57.457478 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:31:01.669351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.668928 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p"] Oct 30 01:31:01.682039 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.673753 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.704856 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.704823 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p"] Oct 30 01:31:01.710281 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.710252 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-eda881-openshift-default-dockercfg-jwbsd\"" Oct 30 01:31:01.812321 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.812280 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.812619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.812592 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzmd\" (UniqueName: \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-kube-api-access-ghzmd\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.812758 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.812630 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.812758 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.812707 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.812758 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.812740 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.812973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.812777 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.812973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.812822 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.813440 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.813305 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.813440 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.813367 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.914654 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.914615 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.914859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.914690 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.914859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.914754 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.914859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.914775 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.915019 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.914899 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.915019 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.914947 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzmd\" (UniqueName: \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-kube-api-access-ghzmd\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.915019 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.914975 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.915166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.915093 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.915166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.915139 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.915166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.915148 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.916031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.915556 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.916031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.915648 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.916031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.915702 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.916031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.915839 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.918044 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.918017 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.918428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.918396 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.938657 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.938569 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.938981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.938954 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzmd\" (UniqueName: \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-kube-api-access-ghzmd\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:01.999285 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:01.999247 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:02.388897 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:02.388853 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p"] Oct 30 01:31:02.396883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:02.396849 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:31:02.397015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:02.396939 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:31:02.397015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:02.396981 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:31:02.570852 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:02.569450 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" event={"ID":"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26","Type":"ContainerStarted","Data":"c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df"} Oct 30 01:31:02.570852 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:02.569522 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" event={"ID":"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26","Type":"ContainerStarted","Data":"e2e334ea1b58654e25054c5c54c5ae58f619ca5bb51da19766049c6f86dafb52"} Oct 30 01:31:03.005353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:03.002301 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:03.005353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:03.002493 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:31:03.005353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:03.002542 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:31:04.000423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:03.999955 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:31:04.000423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:04.000087 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:31:04.999788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:04.999742 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:31:05.000344 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:04.999807 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:31:05.457271 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:05.457234 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:31:05.501684 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:05.501643 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:31:05.548442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:05.548387 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" podStartSLOduration=4.548372063 podStartE2EDuration="4.548372063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:02.64063801 +0000 UTC m=+829.961595050" watchObservedRunningTime="2026-10-30 01:31:05.548372063 +0000 UTC m=+832.869329103" Oct 30 01:31:05.621838 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:05.621809 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:31:06.000602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:06.000564 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:31:06.000983 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:06.000642 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:31:06.999876 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:06.999840 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:31:07.000069 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:06.999902 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:31:07.999902 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:07.999867 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:31:08.000442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:07.999930 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:31:08.999691 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:08.999650 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:31:08.999913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:08.999738 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 01:31:10.004430 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:10.004398 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:10.004883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:10.004781 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:10.005655 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:10.005636 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:31:12.060366 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.060318 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2"] Oct 30 01:31:12.064356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.064336 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.085018 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.084989 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-558327-openshift-default-dockercfg-rmc66\"" Oct 30 01:31:12.097897 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.097856 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2"] Oct 30 01:31:12.180157 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.180115 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.180367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.180167 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.180367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.180283 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.180367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.180321 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.180546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.180445 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.180546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.180505 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qlhl\" (UniqueName: \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-kube-api-access-6qlhl\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.180637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.180617 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.180674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.180651 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.180720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.180672 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282116 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282077 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282153 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282205 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qlhl\" (UniqueName: \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-kube-api-access-6qlhl\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282265 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282301 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282351 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282456 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282497 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282558 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282656 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.282967 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282945 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.283028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.282974 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.283115 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.283094 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.283174 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.283137 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.284798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.284776 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.285226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.285205 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.310042 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.310011 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.311987 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.311926 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qlhl\" (UniqueName: \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-kube-api-access-6qlhl\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-6bdg2\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.378814 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.378780 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:12.708678 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.707156 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2"] Oct 30 01:31:12.711759 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:31:12.711717 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b57a425_bc92_4a04_a643_eb6ae9cc7b44.slice/crio-ba108e1fdcdbd49966ed73cb978b941abcd1afbdb57a2ff9bd102f199fe05625 WatchSource:0}: Error finding container ba108e1fdcdbd49966ed73cb978b941abcd1afbdb57a2ff9bd102f199fe05625: Status 404 returned error can't find the container with id ba108e1fdcdbd49966ed73cb978b941abcd1afbdb57a2ff9bd102f199fe05625 Oct 30 01:31:12.715027 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.714983 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:31:12.715161 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.715083 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:31:12.715161 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:12.715131 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:31:13.671106 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:13.671071 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" event={"ID":"8b57a425-bc92-4a04-a643-eb6ae9cc7b44","Type":"ContainerStarted","Data":"5f0e97e36159149a41343bc88d6810352d504743a5464ad911351bd01c0e70c7"} Oct 30 01:31:13.671543 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:13.671131 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" event={"ID":"8b57a425-bc92-4a04-a643-eb6ae9cc7b44","Type":"ContainerStarted","Data":"ba108e1fdcdbd49966ed73cb978b941abcd1afbdb57a2ff9bd102f199fe05625"} Oct 30 01:31:13.806851 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:13.806791 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" podStartSLOduration=1.806777196 podStartE2EDuration="1.806777196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:13.805161466 +0000 UTC m=+841.126118508" watchObservedRunningTime="2026-10-30 01:31:13.806777196 +0000 UTC m=+841.127734236" Oct 30 01:31:14.383351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:14.380730 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:14.389350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:14.386459 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:31:14.389350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:14.386526 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:31:15.382353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:15.381796 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:31:15.382353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:15.381876 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:31:16.381353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:16.380723 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:31:16.381353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:16.380786 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:31:17.383353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:17.379733 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:31:17.383353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:17.379798 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:31:18.380358 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:18.379756 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:31:18.380358 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:18.379822 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:31:19.229531 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:31:19.229497 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:31:19.380350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:19.379773 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:31:19.380350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:19.379833 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 30 01:31:20.615261 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:20.615229 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:20.616483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:20.616461 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:20.617777 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:20.617735 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 01:31:36.429053 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.428966 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8"] Oct 30 01:31:36.433965 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.433944 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.465631 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.465595 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-803472\"/\"distinct-2-w4-803472-kserve-self-signed-certs\"" Oct 30 01:31:36.469386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.469320 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8"] Oct 30 01:31:36.505165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.504620 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-home\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.505165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.504749 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-tmp-dir\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.505165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.504793 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-dshm\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.505165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.504830 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-model-cache\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.505165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.504864 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zb8f\" (UniqueName: \"kubernetes.io/projected/02cf2cd3-e5cd-4f83-968a-2694debfe092-kube-api-access-4zb8f\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.505165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.504987 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02cf2cd3-e5cd-4f83-968a-2694debfe092-tls-certs\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.606074 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.606033 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-tmp-dir\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.606263 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.606097 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-dshm\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.606263 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.606129 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-model-cache\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.606263 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.606168 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zb8f\" (UniqueName: \"kubernetes.io/projected/02cf2cd3-e5cd-4f83-968a-2694debfe092-kube-api-access-4zb8f\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.606500 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.606263 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02cf2cd3-e5cd-4f83-968a-2694debfe092-tls-certs\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.606500 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.606359 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-home\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.606613 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.606511 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-tmp-dir\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.606736 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.606713 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-home\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.606982 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.606961 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-model-cache\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.610126 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.610063 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-dshm\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.610260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.610175 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02cf2cd3-e5cd-4f83-968a-2694debfe092-tls-certs\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.650400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.650312 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zb8f\" (UniqueName: \"kubernetes.io/projected/02cf2cd3-e5cd-4f83-968a-2694debfe092-kube-api-access-4zb8f\") pod \"distinct-2-w4-803472-kserve-76646db648-xsqr8\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:36.782349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:36.781562 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:37.150922 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:31:37.147734 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02cf2cd3_e5cd_4f83_968a_2694debfe092.slice/crio-c1c33f678d3ad4a3d359d513174b5a0243361ed3c2d6afa9113d5187fff49618 WatchSource:0}: Error finding container c1c33f678d3ad4a3d359d513174b5a0243361ed3c2d6afa9113d5187fff49618: Status 404 returned error can't find the container with id c1c33f678d3ad4a3d359d513174b5a0243361ed3c2d6afa9113d5187fff49618 Oct 30 01:31:37.150922 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:37.150455 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8"] Oct 30 01:31:37.871352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:37.869217 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" event={"ID":"02cf2cd3-e5cd-4f83-968a-2694debfe092","Type":"ContainerStarted","Data":"70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5"} Oct 30 01:31:37.871352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:37.869275 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" event={"ID":"02cf2cd3-e5cd-4f83-968a-2694debfe092","Type":"ContainerStarted","Data":"c1c33f678d3ad4a3d359d513174b5a0243361ed3c2d6afa9113d5187fff49618"} Oct 30 01:31:37.923375 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:37.922684 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podStartSLOduration=1.922661625 podStartE2EDuration="1.922661625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:37.920596207 +0000 UTC m=+865.241553248" watchObservedRunningTime="2026-10-30 01:31:37.922661625 +0000 UTC m=+865.243618667" Oct 30 01:31:38.782135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:38.782098 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:46.782266 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:46.782223 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:46.795624 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:46.795593 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:46.947797 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:31:46.947766 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:31:49.220170 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:31:49.220133 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:32:12.942918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:12.942892 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:32:12.943622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:12.943600 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:32:16.870365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:16.870307 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd"] Oct 30 01:32:16.876128 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:16.876105 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:16.892455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:16.892429 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx"] Oct 30 01:32:16.896532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:16.896513 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:16.906382 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:16.906358 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd"] Oct 30 01:32:16.919986 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:16.919962 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx"] Oct 30 01:32:17.000209 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000180 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.000365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000217 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.000365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000235 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.000365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000302 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.000485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000399 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.000485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000440 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.000547 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000519 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.000625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000559 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.000659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000632 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.000659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000650 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.000720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000673 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4wh7\" (UniqueName: \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-kube-api-access-m4wh7\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.000720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000700 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.000783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000750 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.000819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000812 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.000851 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000839 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j46n4\" (UniqueName: \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-kube-api-access-j46n4\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.000884 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000877 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.000944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000928 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.000997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.000983 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04f7dba-9b73-448b-9c01-c92970e4f715-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.102470 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102439 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.102652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102486 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.102652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102512 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.102652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102566 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.102652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102582 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.102652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102627 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4wh7\" (UniqueName: \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-kube-api-access-m4wh7\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.102923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102668 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.102923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102704 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.102923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102752 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.102923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102793 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j46n4\" (UniqueName: \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-kube-api-access-j46n4\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.102923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102861 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.102923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102887 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.103231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102937 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.103231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.102940 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04f7dba-9b73-448b-9c01-c92970e4f715-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.103231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.103119 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.103231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.103186 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.103231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.103214 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.103514 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.103253 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.103514 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.103407 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.103621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.103594 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04f7dba-9b73-448b-9c01-c92970e4f715-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.103674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.103601 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.104291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.103821 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.104291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.103960 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.104291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.104013 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.104291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.104023 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.104291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.104085 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.104291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.104239 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.104291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.104271 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.107230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.107202 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.107538 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.107515 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.107628 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.107555 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.108053 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.108032 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.124139 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.124079 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.124296 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.124276 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j46n4\" (UniqueName: \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-kube-api-access-j46n4\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.124514 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.124496 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4wh7\" (UniqueName: \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-kube-api-access-m4wh7\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.124698 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.124679 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.192634 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.192605 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:17.214935 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.214910 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:17.519070 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.519032 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd"] Oct 30 01:32:17.530490 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:32:17.530459 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04f7dba_9b73_448b_9c01_c92970e4f715.slice/crio-1ab4a68ecd9d6585458f7c8a2cbd9d9efcb056ed3249ffd3814ef1660e635e73 WatchSource:0}: Error finding container 1ab4a68ecd9d6585458f7c8a2cbd9d9efcb056ed3249ffd3814ef1660e635e73: Status 404 returned error can't find the container with id 1ab4a68ecd9d6585458f7c8a2cbd9d9efcb056ed3249ffd3814ef1660e635e73 Oct 30 01:32:17.542087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.534486 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:17.542087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.534659 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:17.542087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.534742 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:17.555374 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:32:17.555353 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd53b8e7_8d27_49aa_9a7d_21d91a3a6cb6.slice/crio-b3e3bb7ed84fae8868840c73466d4bf77513ef0a97f7dc2a78ef9278ee7a78ef WatchSource:0}: Error finding container b3e3bb7ed84fae8868840c73466d4bf77513ef0a97f7dc2a78ef9278ee7a78ef: Status 404 returned error can't find the container with id b3e3bb7ed84fae8868840c73466d4bf77513ef0a97f7dc2a78ef9278ee7a78ef Oct 30 01:32:17.556475 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.556434 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx"] Oct 30 01:32:17.558171 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.558146 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:17.558245 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.558218 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:17.558285 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:17.558244 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:18.224307 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:18.224262 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" event={"ID":"c04f7dba-9b73-448b-9c01-c92970e4f715","Type":"ContainerStarted","Data":"6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8"} Oct 30 01:32:18.224882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:18.224354 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" event={"ID":"c04f7dba-9b73-448b-9c01-c92970e4f715","Type":"ContainerStarted","Data":"1ab4a68ecd9d6585458f7c8a2cbd9d9efcb056ed3249ffd3814ef1660e635e73"} Oct 30 01:32:18.226966 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:18.226933 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" event={"ID":"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6","Type":"ContainerStarted","Data":"b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe"} Oct 30 01:32:18.227053 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:18.226992 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" event={"ID":"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6","Type":"ContainerStarted","Data":"b3e3bb7ed84fae8868840c73466d4bf77513ef0a97f7dc2a78ef9278ee7a78ef"} Oct 30 01:32:18.304352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:18.303969 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" podStartSLOduration=2.303951185 podStartE2EDuration="2.303951185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:18.288026603 +0000 UTC m=+905.608983643" watchObservedRunningTime="2026-10-30 01:32:18.303951185 +0000 UTC m=+905.624908227" Oct 30 01:32:18.335672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:18.335617 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" podStartSLOduration=2.33560183 podStartE2EDuration="2.33560183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:18.333989453 +0000 UTC m=+905.654946517" watchObservedRunningTime="2026-10-30 01:32:18.33560183 +0000 UTC m=+905.656558870" Oct 30 01:32:19.214354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:19.193312 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:19.214354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:19.203816 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:32:19.214354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:19.203874 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:32:19.221066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:19.221027 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:19.221258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:19.221159 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:32:19.221258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:19.221205 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 01:32:19.247366 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:32:19.247319 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:32:20.198350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:20.194914 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:32:20.198350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:20.194996 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:32:20.217936 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:20.217891 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:32:20.218142 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:20.217981 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 01:32:21.205351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:21.204745 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:32:21.205351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:21.204812 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:32:21.219401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:21.217557 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:32:21.219401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:21.217625 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 01:32:22.210359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:22.209771 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:32:22.210359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:22.209838 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:32:22.227778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:22.227732 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:32:22.227979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:22.227801 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 01:32:23.203111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:23.197536 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:32:23.203111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:23.197599 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:32:23.217435 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:23.217383 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:32:23.217883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:23.217467 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 01:32:24.193214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.193175 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Oct 30 01:32:24.193472 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.193242 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Oct 30 01:32:24.216094 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.216056 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 01:32:24.216265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.216121 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 01:32:24.762992 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.762960 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4"] Oct 30 01:32:24.768913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.768865 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:24.802835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.802803 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4"] Oct 30 01:32:24.883116 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.883080 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:24.891782 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.890382 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgqpv\" (UniqueName: \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-kube-api-access-jgqpv\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:24.892090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.892067 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:24.895432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.895402 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:24.895694 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.895675 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:24.895915 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.895894 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:24.896397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.896373 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:24.896780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.896759 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:24.898357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:24.898314 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.001352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.000600 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.001352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.000653 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.001352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.000702 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.001352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.000794 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.001352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.000877 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.001352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.000912 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgqpv\" (UniqueName: \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-kube-api-access-jgqpv\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.004673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.001899 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.004673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.002601 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.004673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.000993 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.004673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.003610 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.004673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.003642 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.004673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.004002 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.004673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.004256 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.004673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.004529 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.005980 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.005959 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.014907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.014755 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.023341 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.023261 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.024879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.024823 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgqpv\" (UniqueName: \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-kube-api-access-jgqpv\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.120351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.120138 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:25.228934 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.227369 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:25.228934 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.228561 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:25.242025 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.241290 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 01:32:25.290425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.290294 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:25.291450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.291381 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:25.292200 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.292159 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 01:32:25.500477 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.500447 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4"] Oct 30 01:32:25.501034 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:32:25.501011 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf22db55_52cd_47ca_8e8c_6ca9db62c9f1.slice/crio-170fc886da7449f7d3fea0ca5f26bb58fab3dec771b14a1c0cb96aa42ca9ec50 WatchSource:0}: Error finding container 170fc886da7449f7d3fea0ca5f26bb58fab3dec771b14a1c0cb96aa42ca9ec50: Status 404 returned error can't find the container with id 170fc886da7449f7d3fea0ca5f26bb58fab3dec771b14a1c0cb96aa42ca9ec50 Oct 30 01:32:25.503211 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.503171 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:25.503362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.503265 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:25.503362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:25.503319 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:26.343040 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.343004 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz"] Oct 30 01:32:26.347562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.347537 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:26.416692 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.416656 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:32:26.416882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.416708 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:32:26.416882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.416743 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz"] Oct 30 01:32:26.433927 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.433886 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" event={"ID":"df22db55-52cd-47ca-8e8c-6ca9db62c9f1","Type":"ContainerStarted","Data":"94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56"} Oct 30 01:32:26.434111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.433956 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" event={"ID":"df22db55-52cd-47ca-8e8c-6ca9db62c9f1","Type":"ContainerStarted","Data":"170fc886da7449f7d3fea0ca5f26bb58fab3dec771b14a1c0cb96aa42ca9ec50"} Oct 30 01:32:26.436286 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.436262 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6a97281-xstnz\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:26.436471 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.436458 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6a97281-xstnz\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:26.436674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.436656 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwpr2\" (UniqueName: \"kubernetes.io/projected/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-kube-api-access-xwpr2\") pod \"maas-api-revoke-sub-b6a97281-xstnz\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:26.501759 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.501519 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" podStartSLOduration=2.501504475 podStartE2EDuration="2.501504475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:26.49789982 +0000 UTC m=+913.818856860" watchObservedRunningTime="2026-10-30 01:32:26.501504475 +0000 UTC m=+913.822461514" Oct 30 01:32:26.538188 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.538151 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6a97281-xstnz\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:26.538410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.538200 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6a97281-xstnz\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:26.538781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.538751 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwpr2\" (UniqueName: \"kubernetes.io/projected/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-kube-api-access-xwpr2\") pod \"maas-api-revoke-sub-b6a97281-xstnz\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:26.538919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.538891 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6a97281-xstnz\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:26.541169 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.541146 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6a97281-xstnz\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:26.558345 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.558309 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwpr2\" (UniqueName: \"kubernetes.io/projected/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-kube-api-access-xwpr2\") pod \"maas-api-revoke-sub-b6a97281-xstnz\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:26.672674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:26.672585 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:27.019385 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:32:27.019356 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e54b25_6a8c_48e6_8eb8_68ebdd76d1d1.slice/crio-ab10604d1125124ed07159ee3aeaf9753a1cac72e5af5d966d549ce921bc8758 WatchSource:0}: Error finding container ab10604d1125124ed07159ee3aeaf9753a1cac72e5af5d966d549ce921bc8758: Status 404 returned error can't find the container with id ab10604d1125124ed07159ee3aeaf9753a1cac72e5af5d966d549ce921bc8758 Oct 30 01:32:27.019713 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:27.019687 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz"] Oct 30 01:32:27.121081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:27.121037 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:27.122860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:27.122834 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:32:27.122999 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:27.122904 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 01:32:27.442522 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:27.442487 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" event={"ID":"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1","Type":"ContainerStarted","Data":"ab10604d1125124ed07159ee3aeaf9753a1cac72e5af5d966d549ce921bc8758"} Oct 30 01:32:28.120994 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:28.120963 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:32:28.121253 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:28.121228 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 01:32:28.450652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:28.450564 5055 generic.go:356] "Generic (PLEG): container finished" podID="f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1" containerID="90e89f86bb2070acf4862bfbc35c9489f7e3718f975a4e4e83b0c28305416c71" exitCode=0 Oct 30 01:32:28.451065 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:28.450657 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" event={"ID":"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1","Type":"ContainerDied","Data":"90e89f86bb2070acf4862bfbc35c9489f7e3718f975a4e4e83b0c28305416c71"} Oct 30 01:32:29.121738 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:29.121705 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:32:29.121952 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:29.121761 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 01:32:29.823562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:29.823531 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:29.985370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:29.950425 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-xks-ca\") pod \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " Oct 30 01:32:29.985370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:29.950617 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-openshift-ca\") pod \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " Oct 30 01:32:29.985370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:29.950656 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-kube-api-access-xwpr2\" (UniqueName: \"kubernetes.io/projected/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-kube-api-access-xwpr2\") pod \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\" (UID: \"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1\") " Oct 30 01:32:30.016767 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.016668 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-openshift-ca" pod "f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1" (UID: "f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:32:30.081351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.080488 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-xks-ca" pod "f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1" (UID: "f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:32:30.138354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.134518 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-kube-api-access-xwpr2" pod "f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1" (UID: "f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1"). InnerVolumeSpecName "kube-api-access-xwpr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:32:30.138354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.134986 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:32:30.138354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.135037 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 01:32:30.158380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.158235 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-openshift-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:32:30.158380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.158276 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwpr2\" (UniqueName: \"kubernetes.io/projected/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-kube-api-access-xwpr2\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:32:30.158380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.158292 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1-maas-api-xks-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:32:30.532748 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.532630 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" event={"ID":"f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1","Type":"ContainerDied","Data":"ab10604d1125124ed07159ee3aeaf9753a1cac72e5af5d966d549ce921bc8758"} Oct 30 01:32:30.533134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.533080 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab10604d1125124ed07159ee3aeaf9753a1cac72e5af5d966d549ce921bc8758" Oct 30 01:32:30.533433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:30.533361 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz" Oct 30 01:32:31.125473 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:31.125435 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:32:31.126079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:31.126045 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 01:32:32.139495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:32.139455 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:32:32.165859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:32.165421 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 01:32:32.622606 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:32.622564 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 01:32:33.121036 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:33.120995 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 01:32:33.121256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:33.121055 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 01:32:35.162562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:35.162518 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:32:35.163138 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:35.162594 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:35.333410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:35.333376 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:35.334218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:35.334199 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:35.335355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:35.335306 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:32:39.828689 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.828651 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp"] Oct 30 01:32:39.829527 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.829506 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1" containerName="revoke-keys" Oct 30 01:32:39.829527 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.829527 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1" containerName="revoke-keys" Oct 30 01:32:39.829667 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.829651 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1" containerName="revoke-keys" Oct 30 01:32:39.834400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.834379 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:39.877636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.877589 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp"] Oct 30 01:32:39.917879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.917843 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:39.918090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.917898 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:39.918090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.918019 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:39.918195 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.918095 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c56a37-9f9e-4635-96db-82383212e92c-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:39.918241 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.918223 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:39.918320 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.918279 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb9c7\" (UniqueName: \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-kube-api-access-sb9c7\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:39.918408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.918367 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:39.918408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.918388 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:39.918488 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:39.918412 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c56a37-9f9e-4635-96db-82383212e92c-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.036753 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.036710 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.037087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.037057 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.044595 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.044571 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.044856 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.044838 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c56a37-9f9e-4635-96db-82383212e92c-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.045787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.045035 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.046042 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.046014 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb9c7\" (UniqueName: \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-kube-api-access-sb9c7\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.046173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.046062 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.046173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.046084 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.046173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.046123 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c56a37-9f9e-4635-96db-82383212e92c-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.048231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.046938 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.048231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.044452 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.048231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.047547 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c56a37-9f9e-4635-96db-82383212e92c-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.048231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.047792 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.048514 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.044192 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.051727 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.051701 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.054456 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.054395 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c56a37-9f9e-4635-96db-82383212e92c-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.074377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.072583 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb9c7\" (UniqueName: \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-kube-api-access-sb9c7\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.074377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.073961 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.148623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.148545 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:40.473899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.473865 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp"] Oct 30 01:32:40.474899 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:32:40.474871 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c56a37_9f9e_4635_96db_82383212e92c.slice/crio-cd2dec5afc4b1e90e2a3322e05301e8bf5c55deb306bae6a1a353d36a4d843a7 WatchSource:0}: Error finding container cd2dec5afc4b1e90e2a3322e05301e8bf5c55deb306bae6a1a353d36a4d843a7: Status 404 returned error can't find the container with id cd2dec5afc4b1e90e2a3322e05301e8bf5c55deb306bae6a1a353d36a4d843a7 Oct 30 01:32:40.482655 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.477857 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:40.482655 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.477946 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:40.482655 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.477994 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:40.662845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.662814 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" event={"ID":"09c56a37-9f9e-4635-96db-82383212e92c","Type":"ContainerStarted","Data":"2e8ebf104e40759747dfff4bdef8cad508adf388cf0b804683a3bc86df4c46a9"} Oct 30 01:32:40.663032 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.662870 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" event={"ID":"09c56a37-9f9e-4635-96db-82383212e92c","Type":"ContainerStarted","Data":"cd2dec5afc4b1e90e2a3322e05301e8bf5c55deb306bae6a1a353d36a4d843a7"} Oct 30 01:32:40.711404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.711352 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" podStartSLOduration=1.7113180350000001 podStartE2EDuration="1.711318035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:40.709046024 +0000 UTC m=+928.030003065" watchObservedRunningTime="2026-10-30 01:32:40.711318035 +0000 UTC m=+928.032275073" Oct 30 01:32:40.716961 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.716925 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz"] Oct 30 01:32:40.721106 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.721074 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.748928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.748898 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz"] Oct 30 01:32:40.755678 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.753312 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr"] Oct 30 01:32:40.761408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.761381 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.785380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.785293 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr"] Oct 30 01:32:40.857611 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.857559 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.857611 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.857609 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beafde80-c491-4977-8a5e-8abe3827fb07-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.858167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.857768 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.858167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.857813 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.858167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.857839 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hdx6\" (UniqueName: \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-kube-api-access-4hdx6\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.858167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.857900 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.858167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.857930 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beafde80-c491-4977-8a5e-8abe3827fb07-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.858167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.857970 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.858167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.857994 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.858167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.858076 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.858167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.858107 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.858167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.858158 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.858716 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.858201 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.858716 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.858509 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.858716 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.858666 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.858716 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.858711 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbnf\" (UniqueName: \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-kube-api-access-8hbnf\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.858937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.858736 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.858937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.858800 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.959841 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.959805 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.959995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.959855 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbnf\" (UniqueName: \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-kube-api-access-8hbnf\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.959995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.959893 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.959995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.959980 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.960218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960016 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.960218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960044 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beafde80-c491-4977-8a5e-8abe3827fb07-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.960218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960087 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.960218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960122 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.960218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960147 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hdx6\" (UniqueName: \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-kube-api-access-4hdx6\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.960218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960213 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.960548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960245 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beafde80-c491-4977-8a5e-8abe3827fb07-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.960548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960284 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.960548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960307 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.960548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960436 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.960548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960484 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.960548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960542 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.960831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960569 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.960831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960613 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.960831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960625 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.961057 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.960545 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.961348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.961303 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.961495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.961377 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.961679 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.961658 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.961754 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.961739 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beafde80-c491-4977-8a5e-8abe3827fb07-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.961866 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.961845 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.961940 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.961898 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.961992 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.961959 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.962153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.962131 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.963005 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.962976 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.963299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.963283 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.964217 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.964202 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.964285 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.964248 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beafde80-c491-4977-8a5e-8abe3827fb07-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.980731 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.980645 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.981959 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.981937 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbnf\" (UniqueName: \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-kube-api-access-8hbnf\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:40.982078 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.981934 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hdx6\" (UniqueName: \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-kube-api-access-4hdx6\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:40.982078 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:40.981938 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:41.039352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.035573 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:41.080485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.080190 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:41.150363 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.148851 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:41.153391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.153216 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:32:41.153391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.153263 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:32:41.395735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.395703 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz"] Oct 30 01:32:41.399184 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:32:41.399149 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52115e42_4083_4b2a_9e5d_0ac4b0a139f0.slice/crio-1d663bf3e8feb1ccc25fc48fe5798557ffa5a36cbf864310eacd8083f08b4780 WatchSource:0}: Error finding container 1d663bf3e8feb1ccc25fc48fe5798557ffa5a36cbf864310eacd8083f08b4780: Status 404 returned error can't find the container with id 1d663bf3e8feb1ccc25fc48fe5798557ffa5a36cbf864310eacd8083f08b4780 Oct 30 01:32:41.401882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.401852 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:41.401990 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.401917 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:41.401990 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.401961 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:41.455178 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.455148 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr"] Oct 30 01:32:41.461289 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:32:41.461259 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeafde80_c491_4977_8a5e_8abe3827fb07.slice/crio-9b61e471eedf38b5eb03f696abde707c0d9a8443efd54f7c5e250c2b21807e7c WatchSource:0}: Error finding container 9b61e471eedf38b5eb03f696abde707c0d9a8443efd54f7c5e250c2b21807e7c: Status 404 returned error can't find the container with id 9b61e471eedf38b5eb03f696abde707c0d9a8443efd54f7c5e250c2b21807e7c Oct 30 01:32:41.478016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.472265 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:41.478016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.472371 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:41.478016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.472418 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:41.723744 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.722868 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" event={"ID":"52115e42-4083-4b2a-9e5d-0ac4b0a139f0","Type":"ContainerStarted","Data":"3cc3494dfb91177b5f35acd69f063b4151f3c3c397a5138b638f8661c7d0a668"} Oct 30 01:32:41.723744 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.722933 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" event={"ID":"52115e42-4083-4b2a-9e5d-0ac4b0a139f0","Type":"ContainerStarted","Data":"1d663bf3e8feb1ccc25fc48fe5798557ffa5a36cbf864310eacd8083f08b4780"} Oct 30 01:32:41.728771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.728733 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" event={"ID":"beafde80-c491-4977-8a5e-8abe3827fb07","Type":"ContainerStarted","Data":"b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981"} Oct 30 01:32:41.728937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.728781 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" event={"ID":"beafde80-c491-4977-8a5e-8abe3827fb07","Type":"ContainerStarted","Data":"9b61e471eedf38b5eb03f696abde707c0d9a8443efd54f7c5e250c2b21807e7c"} Oct 30 01:32:41.826573 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.822632 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podStartSLOduration=1.8226114519999999 podStartE2EDuration="1.822611452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:41.772472349 +0000 UTC m=+929.093429390" watchObservedRunningTime="2026-10-30 01:32:41.822611452 +0000 UTC m=+929.143568496" Oct 30 01:32:41.844709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:41.844645 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podStartSLOduration=1.844625158 podStartE2EDuration="1.844625158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:41.836046758 +0000 UTC m=+929.157003798" watchObservedRunningTime="2026-10-30 01:32:41.844625158 +0000 UTC m=+929.165582201" Oct 30 01:32:42.039412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:42.038412 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:42.050571 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:42.050123 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:32:42.050571 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:42.050211 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:32:42.094351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:42.093387 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:42.094351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:42.093566 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:32:42.094351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:42.093629 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:32:42.149080 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:42.149043 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:32:42.149245 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:42.149108 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:32:43.038233 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.038197 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:32:43.038766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.038735 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:32:43.090486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.090448 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:32:43.091073 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.091027 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:32:43.246408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.242731 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:32:43.247267 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.247232 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:32:43.569363 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.568451 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:43.619353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.618417 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:43.643350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.642668 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:32:43.758354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.757917 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:32:43.841510 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.826488 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:43.944353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:43.940413 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:44.036805 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:44.036768 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:32:44.037114 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:44.037085 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:32:44.085603 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:44.085559 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:32:44.086301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:44.085829 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:32:44.153438 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:44.149682 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:32:44.153438 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:44.149742 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:32:45.079421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:45.078973 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:32:45.079421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:45.079045 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:32:45.119353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:45.095429 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:32:45.119353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:45.095491 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:32:45.149265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:45.149150 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:32:45.149265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:45.149206 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:32:46.040614 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:46.040573 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:32:46.041171 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:46.041142 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:32:46.096473 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:46.096433 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:32:46.096750 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:46.096721 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:32:46.155799 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:46.155754 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:32:46.156053 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:46.156025 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:32:47.038384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.038346 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:32:47.038700 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.038665 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:32:47.091354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.082408 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:32:47.091354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.082481 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:32:47.154482 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.154451 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:32:47.154846 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.154732 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:32:47.749173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.748594 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s"] Oct 30 01:32:47.754175 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.754148 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:47.782105 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.780610 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"distinct-2-w0-a3a2ac-kserve-self-signed-certs\"" Oct 30 01:32:47.851526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.826348 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s"] Oct 30 01:32:47.860162 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.860126 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-tmp-dir\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:47.860442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.860421 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-home\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:47.860633 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.860614 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpxw\" (UniqueName: \"kubernetes.io/projected/cfb150d4-d9ce-4574-b245-0248f5f4062f-kube-api-access-6lpxw\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:47.860752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.860737 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb150d4-d9ce-4574-b245-0248f5f4062f-tls-certs\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:47.860912 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.860896 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-dshm\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:47.861101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.861085 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-model-cache\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:47.976852 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:47.962102 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-model-cache\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.051408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.047313 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-model-cache\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.051408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.047407 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-tmp-dir\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.051408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.047560 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-home\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.051943 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.051895 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpxw\" (UniqueName: \"kubernetes.io/projected/cfb150d4-d9ce-4574-b245-0248f5f4062f-kube-api-access-6lpxw\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.052163 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.052139 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb150d4-d9ce-4574-b245-0248f5f4062f-tls-certs\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.062596 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.062559 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-dshm\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.143544 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.061717 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-tmp-dir\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.144187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.081375 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-home\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.144341 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.057571 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:32:48.144709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.144584 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:32:48.144945 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.106754 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:32:48.145224 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.145185 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:32:48.156959 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.141002 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb150d4-d9ce-4574-b245-0248f5f4062f-tls-certs\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.157204 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.156754 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:32:48.157570 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.157448 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 01:32:48.179831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.179760 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-dshm\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.247508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.247371 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpxw\" (UniqueName: \"kubernetes.io/projected/cfb150d4-d9ce-4574-b245-0248f5f4062f-kube-api-access-6lpxw\") pod \"distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.393123 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.393032 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:48.868009 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:48.867975 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s"] Oct 30 01:32:48.890093 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:32:48.890052 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcfb150d4_d9ce_4574_b245_0248f5f4062f.slice/crio-4c9ad9f75d9a4f84d088c59e8475d4b127ec800299f99aa5f0623c627da01b61 WatchSource:0}: Error finding container 4c9ad9f75d9a4f84d088c59e8475d4b127ec800299f99aa5f0623c627da01b61: Status 404 returned error can't find the container with id 4c9ad9f75d9a4f84d088c59e8475d4b127ec800299f99aa5f0623c627da01b61 Oct 30 01:32:49.036707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.036673 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:32:49.036873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.036731 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 01:32:49.081371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.081209 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:32:49.081371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.081272 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" Oct 30 01:32:49.145239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.145204 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" event={"ID":"cfb150d4-d9ce-4574-b245-0248f5f4062f","Type":"ContainerStarted","Data":"0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c"} Oct 30 01:32:49.145239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.145245 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" event={"ID":"cfb150d4-d9ce-4574-b245-0248f5f4062f","Type":"ContainerStarted","Data":"4c9ad9f75d9a4f84d088c59e8475d4b127ec800299f99aa5f0623c627da01b61"} Oct 30 01:32:49.194109 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.193998 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podStartSLOduration=2.193983881 podStartE2EDuration="2.193983881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:49.191301872 +0000 UTC m=+936.512258914" watchObservedRunningTime="2026-10-30 01:32:49.193983881 +0000 UTC m=+936.514940922" Oct 30 01:32:49.245212 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:32:49.245170 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:32:49.277982 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.277951 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:49.278510 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.278444 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:49.281211 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.281185 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:32:49.402528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:49.401489 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:50.617352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:50.616415 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:50.812365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:50.811739 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:50.839490 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:50.839447 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.042151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:51.042102 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:32:51.042472 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:51.042448 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.100558 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:51.100477 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:32:51.100871 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:51.100801 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.225362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:51.219777 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:51.225362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:51.221738 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:51.229648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:51.229621 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:32:52.108493 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:52.108453 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:32:52.109146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:52.109094 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.566104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:52.566061 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.860234 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:52.850452 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.860234 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:52.859450 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 01:32:52.860234 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:52.859889 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.943910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:52.943462 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.171949 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.171860 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:32:53.195797 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.195759 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.571582 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.571536 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.572294 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.572270 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:32:53.769659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.769616 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.819861 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.819819 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.874090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.873986 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 01:32:53.877279 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.877237 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 01:32:53.878556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.878494 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:32:53.878811 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.878794 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:32:53.941126 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.941083 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.973706 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.973657 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.974834 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:53.974810 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:32:54.270353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:54.245637 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:32:54.270353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:54.245693 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:54.270353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:54.253520 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:32:54.370355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:54.370288 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:32:54.394408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:54.394112 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:32:54.458592 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:54.458563 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:32:54.927407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:54.926574 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:54.927622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:54.927507 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:54.928389 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:54.928366 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:32:57.375119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.375076 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr"] Oct 30 01:32:57.380648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.380621 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.402028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.401999 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr"] Oct 30 01:32:57.416231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.416203 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5"] Oct 30 01:32:57.420462 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.420436 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.436830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.436803 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5"] Oct 30 01:32:57.516311 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516280 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.516526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516316 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b66c7d3-7336-4beb-af15-3c013ec43512-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.516526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516391 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.516526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516421 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.516709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516557 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.516709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516603 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.516709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516647 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.516877 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516782 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smtkx\" (UniqueName: \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-kube-api-access-smtkx\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.516877 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516810 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.516877 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516835 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6x8v\" (UniqueName: \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-kube-api-access-j6x8v\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.516993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516954 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.516993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516976 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.517089 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.516992 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.517149 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.517130 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.517212 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.517163 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.517212 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.517193 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.517284 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.517234 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.517284 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.517260 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.618581 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.618547 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.618581 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.618583 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b66c7d3-7336-4beb-af15-3c013ec43512-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.618832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.618618 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.618832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.618656 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.618832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.618694 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.618832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.618718 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.618832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.618763 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.619104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.618854 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smtkx\" (UniqueName: \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-kube-api-access-smtkx\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.619104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.618881 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.619104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.618919 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6x8v\" (UniqueName: \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-kube-api-access-j6x8v\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.619104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619004 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.619104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619038 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.619104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619063 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.619450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619142 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.619450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619152 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.619450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619176 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.619450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619196 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.619450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619204 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.619450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619250 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.619450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619288 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.619450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619426 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b66c7d3-7336-4beb-af15-3c013ec43512-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.619890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619450 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.619890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619618 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.619890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.619766 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.620214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.620118 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.620214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.620171 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.620453 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.620431 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.620543 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.620438 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.621799 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.621778 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.621995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.621979 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.622144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.622129 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.622676 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.622656 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.639607 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.639523 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6x8v\" (UniqueName: \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-kube-api-access-j6x8v\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.639764 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.639741 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-84xp5\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:57.643573 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.643541 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.643868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.643852 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smtkx\" (UniqueName: \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-kube-api-access-smtkx\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-8lpmr\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.696386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.696351 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:57.740019 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:57.739979 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:58.022009 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:32:58.021980 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b66c7d3_7336_4beb_af15_3c013ec43512.slice/crio-c4b7e25fbbad4cc4d795138766943354585da455438ca0dd2bccca18f7a46a77 WatchSource:0}: Error finding container c4b7e25fbbad4cc4d795138766943354585da455438ca0dd2bccca18f7a46a77: Status 404 returned error can't find the container with id c4b7e25fbbad4cc4d795138766943354585da455438ca0dd2bccca18f7a46a77 Oct 30 01:32:58.023685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.023661 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr"] Oct 30 01:32:58.024689 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.024655 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:58.024783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.024734 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:58.024845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.024780 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:58.107078 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:32:58.107046 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd262c4_5d37_4cd2_85fe_7e3bbb053435.slice/crio-cf0212fa38072fefe32d997696477dcd647f0a81d36f21c9911ca466cb6091b4 WatchSource:0}: Error finding container cf0212fa38072fefe32d997696477dcd647f0a81d36f21c9911ca466cb6091b4: Status 404 returned error can't find the container with id cf0212fa38072fefe32d997696477dcd647f0a81d36f21c9911ca466cb6091b4 Oct 30 01:32:58.107753 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.107730 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5"] Oct 30 01:32:58.109863 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.109828 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:58.109964 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.109945 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:58.110016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.110000 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:32:58.396370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.394241 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:58.413905 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.413875 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:58.513568 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.513521 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" event={"ID":"7b66c7d3-7336-4beb-af15-3c013ec43512","Type":"ContainerStarted","Data":"80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8"} Oct 30 01:32:58.513766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.513594 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" event={"ID":"7b66c7d3-7336-4beb-af15-3c013ec43512","Type":"ContainerStarted","Data":"c4b7e25fbbad4cc4d795138766943354585da455438ca0dd2bccca18f7a46a77"} Oct 30 01:32:58.516821 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.516793 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" event={"ID":"1bd262c4-5d37-4cd2-85fe-7e3bbb053435","Type":"ContainerStarted","Data":"06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95"} Oct 30 01:32:58.517004 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.516986 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" event={"ID":"1bd262c4-5d37-4cd2-85fe-7e3bbb053435","Type":"ContainerStarted","Data":"cf0212fa38072fefe32d997696477dcd647f0a81d36f21c9911ca466cb6091b4"} Oct 30 01:32:58.528779 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.528752 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:32:58.558288 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.558242 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" podStartSLOduration=1.5582290429999999 podStartE2EDuration="1.558229043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:58.55563718 +0000 UTC m=+945.876594221" watchObservedRunningTime="2026-10-30 01:32:58.558229043 +0000 UTC m=+945.879186148" Oct 30 01:32:58.670511 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.668820 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" podStartSLOduration=1.6687942009999999 podStartE2EDuration="1.668794201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:58.644204859 +0000 UTC m=+945.965161899" watchObservedRunningTime="2026-10-30 01:32:58.668794201 +0000 UTC m=+945.989751241" Oct 30 01:32:58.698351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.697861 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:32:58.740811 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:58.740775 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:32:59.703351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:59.703154 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:32:59.703351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:59.703227 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:32:59.741494 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:59.741464 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-84xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:32:59.741652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:32:59.741515 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 01:33:00.699185 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:00.698711 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:33:00.699185 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:00.698779 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:33:00.741022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:00.740990 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-84xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:33:00.741585 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:00.741548 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 01:33:01.697351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:01.697107 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:33:01.697351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:01.697181 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:33:01.740896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:01.740597 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-84xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:33:01.740896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:01.740651 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 01:33:02.703189 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:02.701853 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:33:02.703189 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:02.701915 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:33:02.740682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:02.740644 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-84xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:33:02.740860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:02.740702 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 01:33:03.699350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:03.698739 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:33:03.699350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:03.698816 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:33:03.749438 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:03.749386 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-84xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:33:03.749645 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:03.749463 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 01:33:04.696783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:04.696692 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 01:33:04.696783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:04.696747 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 01:33:04.740552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:04.740517 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-84xp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 30 01:33:04.741056 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:04.740566 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 30 01:33:05.734112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:05.734078 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:33:05.735308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:05.735279 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:33:05.736321 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:05.736298 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:33:05.856170 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:05.856080 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:33:05.858135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:05.858102 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:33:05.865316 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:05.860523 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:33:13.381355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:13.380696 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.402632 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:13.402592 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.467563 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:13.467516 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.689412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:13.671681 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.762461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:13.762415 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.798477 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:13.798435 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.937474 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:13.937429 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded" Oct 30 01:33:14.256581 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.256432 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.387355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.385273 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.387355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.386086 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:33:14.423509 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.423436 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.424162 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.424140 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:33:14.573429 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.489947 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 01:33:14.573429 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.490401 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:33:14.660106 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.657428 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 01:33:14.687862 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.686730 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:33:14.688520 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.688497 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:33:14.797178 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.775394 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:33:14.798276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.797807 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:33:14.829220 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.828172 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.829220 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.828765 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:33:14.975375 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:14.949751 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.138404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:15.113783 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:33:15.260351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:15.257936 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.260351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:15.260106 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:33:15.394450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:15.386454 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.451351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:15.426798 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.678531 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:15.678433 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.945411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:15.697669 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 01:33:15.945411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:15.926904 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:33:15.947278 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:15.864480 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.982679 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:15.950524 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:16.095258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.095225 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:33:16.271510 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.253647 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:33:16.310473 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.310424 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:16.423426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.421404 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:16.496362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.452295 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:16.543371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.543008 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:33:16.580355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.564226 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:33:16.581191 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.581164 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:33:16.581452 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.581438 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:33:16.581621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.581609 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:33:16.608094 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.608065 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:33:16.608375 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:16.608361 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:33:17.397402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:17.394006 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:33:19.220697 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:33:19.220664 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:33:24.952720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:24.952688 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g"] Oct 30 01:33:24.958614 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:24.958595 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:24.997780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:24.997752 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g"] Oct 30 01:33:25.029177 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.029138 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.029379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.029227 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.029379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.029265 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvjt\" (UniqueName: \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-kube-api-access-6kvjt\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.029379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.029304 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.029587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.029475 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9186ef0-4662-4afc-86a5-27d290426e41-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.029587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.029513 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.029695 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.029623 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.029793 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.029773 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.029842 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.029825 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9186ef0-4662-4afc-86a5-27d290426e41-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.131218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.131170 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.131433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.131232 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvjt\" (UniqueName: \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-kube-api-access-6kvjt\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.131433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.131296 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.131433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.131375 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9186ef0-4662-4afc-86a5-27d290426e41-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.131433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.131410 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.131670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.131477 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.131670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.131580 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.131670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.131626 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9186ef0-4662-4afc-86a5-27d290426e41-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.131823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.131740 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.132254 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.132224 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9186ef0-4662-4afc-86a5-27d290426e41-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.132531 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.132503 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.132735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.132555 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.132831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.132783 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.133392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.133369 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.144713 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.144683 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.144908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.144884 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9186ef0-4662-4afc-86a5-27d290426e41-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.152416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.152361 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.152560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.152536 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvjt\" (UniqueName: \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-kube-api-access-6kvjt\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.294185 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.294106 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:25.628509 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.628480 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g"] Oct 30 01:33:25.628647 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:33:25.628577 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9186ef0_4662_4afc_86a5_27d290426e41.slice/crio-07c6bad224a3c760287d11905e731d667b8d8bd3ff23efcba72a7e7af35d8601 WatchSource:0}: Error finding container 07c6bad224a3c760287d11905e731d667b8d8bd3ff23efcba72a7e7af35d8601: Status 404 returned error can't find the container with id 07c6bad224a3c760287d11905e731d667b8d8bd3ff23efcba72a7e7af35d8601 Oct 30 01:33:25.631903 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.631870 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:33:25.631994 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.631938 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:33:25.631994 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.631970 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:33:25.668820 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:25.668684 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" event={"ID":"e9186ef0-4662-4afc-86a5-27d290426e41","Type":"ContainerStarted","Data":"07c6bad224a3c760287d11905e731d667b8d8bd3ff23efcba72a7e7af35d8601"} Oct 30 01:33:26.690350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:26.688694 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" event={"ID":"e9186ef0-4662-4afc-86a5-27d290426e41","Type":"ContainerStarted","Data":"f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3"} Oct 30 01:33:26.747280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:26.747230 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" podStartSLOduration=2.747215769 podStartE2EDuration="2.747215769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:26.744588159 +0000 UTC m=+974.065545202" watchObservedRunningTime="2026-10-30 01:33:26.747215769 +0000 UTC m=+974.068172806" Oct 30 01:33:27.294837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:27.294799 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:27.296736 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:27.296696 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:33:27.296873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:27.296755 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 01:33:28.295152 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:28.295115 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:33:28.295601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:28.295198 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 01:33:29.298768 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:29.298734 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:33:29.299315 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:29.298812 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 01:33:30.294693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:30.294419 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:33:30.294693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:30.294475 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 01:33:31.295353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:31.294538 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:33:31.295353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:31.294604 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 01:33:32.295031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:32.294991 5055 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 30 01:33:32.298000 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:32.295060 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 30 01:33:33.223558 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.223526 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6"] Oct 30 01:33:33.227998 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.227975 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.251205 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.251177 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6"] Oct 30 01:33:33.299912 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.299878 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:33.300758 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.300735 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:33.301464 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.301443 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:33:33.312552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.312530 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.312676 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.312562 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.312721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.312693 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.312767 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.312725 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222113ca-8659-476a-97a0-891ed690141e-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.312844 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.312831 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.312918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.312879 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.313031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.313011 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd758\" (UniqueName: \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-kube-api-access-vd758\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.313118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.313094 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222113ca-8659-476a-97a0-891ed690141e-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.313192 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.313130 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.414566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.414530 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.414766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.414594 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.414766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.414648 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd758\" (UniqueName: \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-kube-api-access-vd758\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.414766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.414688 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222113ca-8659-476a-97a0-891ed690141e-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.414766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.414717 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.415003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.414833 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.415003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.414861 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.415003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.414952 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.415003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.414981 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222113ca-8659-476a-97a0-891ed690141e-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.415708 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.415681 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.415876 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.415849 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222113ca-8659-476a-97a0-891ed690141e-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.416318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.416298 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.416808 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.416790 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.417272 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.417254 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.418282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.418255 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222113ca-8659-476a-97a0-891ed690141e-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.426919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.426871 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.434992 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.434976 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.442895 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.442867 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd758\" (UniqueName: \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-kube-api-access-vd758\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.542810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.542734 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:33.899802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.895681 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6"] Oct 30 01:33:33.905571 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:33:33.905539 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222113ca_8659_476a_97a0_891ed690141e.slice/crio-e8bbd309e27f11e17759878339da6307e1fa00c472efbd386b637d9865880a61 WatchSource:0}: Error finding container e8bbd309e27f11e17759878339da6307e1fa00c472efbd386b637d9865880a61: Status 404 returned error can't find the container with id e8bbd309e27f11e17759878339da6307e1fa00c472efbd386b637d9865880a61 Oct 30 01:33:33.907784 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.907747 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:33:33.907882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.907819 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:33:33.907882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:33.907847 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:33:34.787349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:34.787000 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" event={"ID":"222113ca-8659-476a-97a0-891ed690141e","Type":"ContainerStarted","Data":"1e6813fa5a0dc96131839cbf0265a63e3e7f339f55130bb7b7569dcf5d2d9ead"} Oct 30 01:33:34.787349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:34.787063 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" event={"ID":"222113ca-8659-476a-97a0-891ed690141e","Type":"ContainerStarted","Data":"e8bbd309e27f11e17759878339da6307e1fa00c472efbd386b637d9865880a61"} Oct 30 01:33:34.855937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:34.855890 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podStartSLOduration=1.85587714 podStartE2EDuration="1.85587714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:34.841157512 +0000 UTC m=+982.162114552" watchObservedRunningTime="2026-10-30 01:33:34.85587714 +0000 UTC m=+982.176834180" Oct 30 01:33:35.543154 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:35.543123 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:35.544807 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:35.544784 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:33:35.544941 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:35.544834 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 01:33:36.543505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:36.543462 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:33:36.543924 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:36.543527 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 01:33:37.575354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:37.558253 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:33:37.575354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:37.558360 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 01:33:38.548737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:38.548696 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:33:38.549868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:38.549835 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 01:33:38.834518 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:38.834416 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:38.935782 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:38.935742 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:39.015480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.015435 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:39.081356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.080365 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Oct 30 01:33:39.316281 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.316228 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 01:33:39.426066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.426028 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:39.544064 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.544031 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:33:39.544382 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.544352 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 01:33:39.597174 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.597076 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds"] Oct 30 01:33:39.608824 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.608802 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:39.625317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.624176 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ec2345-serving-cert\"" Oct 30 01:33:39.683431 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.649804 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds"] Oct 30 01:33:39.754470 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.754434 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-maas-api-tls\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:39.758482 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.758450 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-metrics-certs\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:39.766351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.762450 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcbqt\" (UniqueName: \"kubernetes.io/projected/43cb60a1-5ab7-45b9-b244-13600d3bee3c-kube-api-access-kcbqt\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:39.807465 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.807426 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb60a1-5ab7-45b9-b244-13600d3bee3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:39.953400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.946934 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb60a1-5ab7-45b9-b244-13600d3bee3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:39.953400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.947026 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-maas-api-tls\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:39.953400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.947093 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-metrics-certs\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:39.953400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.947178 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcbqt\" (UniqueName: \"kubernetes.io/projected/43cb60a1-5ab7-45b9-b244-13600d3bee3c-kube-api-access-kcbqt\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:39.953400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:39.948214 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb60a1-5ab7-45b9-b244-13600d3bee3c-openshift-service-ca\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:40.033962 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:40.033932 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-metrics-certs\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:40.114380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:40.107855 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-maas-api-tls\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:40.125352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:40.122430 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcbqt\" (UniqueName: \"kubernetes.io/projected/43cb60a1-5ab7-45b9-b244-13600d3bee3c-kube-api-access-kcbqt\") pod \"maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:40.312409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:40.301623 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:40.334355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:40.319461 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 01:33:40.334355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:40.328693 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:33:40.546236 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:40.546206 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:33:40.557102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:40.546503 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 01:33:41.102654 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:41.049349 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:41.349353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:41.347365 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:41.403194 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:41.403089 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:41.532775 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:41.532721 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds"] Oct 30 01:33:41.553078 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:41.553046 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:33:41.553369 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:41.553317 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 01:33:41.649516 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:41.649486 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:33:42.071407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:42.051428 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:42.071407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:42.067191 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:33:42.356480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:42.355978 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:42.397484 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:42.397446 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" event={"ID":"43cb60a1-5ab7-45b9-b244-13600d3bee3c","Type":"ContainerStarted","Data":"c3af45edc50021bcbf73068f632a0b155bba52e71d11d1fe66d448a1b04c9425"} Oct 30 01:33:42.410540 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:42.410479 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:33:42.411294 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:42.411258 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:33:42.537352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:42.501031 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:42.624457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:42.597892 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:33:42.624457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:42.597953 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 01:33:42.782460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:42.782428 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:33:43.005365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:43.005284 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:43.051887 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:43.051753 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:43.113614 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:43.112441 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:33:43.114591 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:43.114566 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:33:43.408207 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:43.408120 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" event={"ID":"43cb60a1-5ab7-45b9-b244-13600d3bee3c","Type":"ContainerStarted","Data":"6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15"} Oct 30 01:33:43.412513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:43.412489 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:33:43.450154 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:43.450098 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" podStartSLOduration=3.260659088 podStartE2EDuration="4.450083739s" totalImagesPullingTime="1.189424651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:43.447904421 +0000 UTC m=+990.768861461" watchObservedRunningTime="2026-10-30 01:33:43.450083739 +0000 UTC m=+990.771040809" Oct 30 01:33:43.544183 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:43.544142 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 30 01:33:43.544393 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:43.544209 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 30 01:33:44.547862 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:44.547831 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:44.548314 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:44.548286 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:44.548958 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:44.548941 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:33:49.228687 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:33:49.228644 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:33:49.421617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:33:49.421587 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:34:12.721079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.721039 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj"] Oct 30 01:34:12.728122 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.727595 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.747530 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.747503 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a06822-openshift-default-dockercfg-f65v4\"" Oct 30 01:34:12.772350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.772155 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj"] Oct 30 01:34:12.808528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.808487 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c97571-2fe7-4f32-bade-7a725d5fce15-istiod-ca-cert\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.808818 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.808801 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-certs\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.808934 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.808919 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-credential-socket\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.809035 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.809023 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-socket\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.809142 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.809129 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-token\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.809282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.809269 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jrzg\" (UniqueName: \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-kube-api-access-5jrzg\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.809412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.809398 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-envoy\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.809529 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.809508 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-podinfo\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.809729 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.809716 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-data\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.962350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.953012 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-data\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.962350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.953156 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c97571-2fe7-4f32-bade-7a725d5fce15-istiod-ca-cert\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.962350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.953243 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-certs\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.962350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.953277 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-credential-socket\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.962350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.953313 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-socket\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.962350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.953380 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-token\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.962350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.953497 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jrzg\" (UniqueName: \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-kube-api-access-5jrzg\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.962350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.953539 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-envoy\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.962350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.953572 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-podinfo\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.962350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.953629 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-data\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.981446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.966616 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-credential-socket\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.981446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.967032 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c97571-2fe7-4f32-bade-7a725d5fce15-istiod-ca-cert\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.981446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.967853 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-podinfo\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.981446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.972913 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-certs\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.981446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.973129 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-socket\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.993347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.987978 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-token\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.993347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.988354 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-envoy\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:12.993347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:12.990823 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jrzg\" (UniqueName: \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-kube-api-access-5jrzg\") pod \"e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:13.071350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.070663 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a06822-openshift-default-dockercfg-f65v4\"" Oct 30 01:34:13.071350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.071237 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:13.077549 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.077409 5055 scope.go:122] "RemoveContainer" containerID="f289ba428e7741739aa4fd4b07972b76ff66b451252842d2efbe4b4ad647e7a4" Oct 30 01:34:13.110393 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.110254 5055 scope.go:122] "RemoveContainer" containerID="28a92db128cddcda71a44757b3e0039cf3e86c696e6fd6f1d2ceeea9f7bc2f50" Oct 30 01:34:13.141126 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.141098 5055 scope.go:122] "RemoveContainer" containerID="704a485ef7351f302d5e092303d8823fcb2f5769a410d1a5832139bf382ea13b" Oct 30 01:34:13.155292 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.155266 5055 scope.go:122] "RemoveContainer" containerID="5b51e0e6d82e64f74d90ffc8ab75a70088b26bceb81b801725fa91a7e8b6d36e" Oct 30 01:34:13.428426 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:34:13.428395 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c97571_2fe7_4f32_bade_7a725d5fce15.slice/crio-8a42fef3dc6b6bd3b6349fb9893e0927b17045a4292425ca777ca5cb0552fb3b WatchSource:0}: Error finding container 8a42fef3dc6b6bd3b6349fb9893e0927b17045a4292425ca777ca5cb0552fb3b: Status 404 returned error can't find the container with id 8a42fef3dc6b6bd3b6349fb9893e0927b17045a4292425ca777ca5cb0552fb3b Oct 30 01:34:13.429975 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.429950 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj"] Oct 30 01:34:13.431270 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.431236 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:13.431418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.431311 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:13.431418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.431395 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:13.646242 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.646190 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" event={"ID":"86c97571-2fe7-4f32-bade-7a725d5fce15","Type":"ContainerStarted","Data":"8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab"} Oct 30 01:34:13.646242 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.646248 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" event={"ID":"86c97571-2fe7-4f32-bade-7a725d5fce15","Type":"ContainerStarted","Data":"8a42fef3dc6b6bd3b6349fb9893e0927b17045a4292425ca777ca5cb0552fb3b"} Oct 30 01:34:13.691482 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:13.691423 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" podStartSLOduration=1.69140512 podStartE2EDuration="1.69140512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:13.687302538 +0000 UTC m=+1021.008259579" watchObservedRunningTime="2026-10-30 01:34:13.69140512 +0000 UTC m=+1021.012362162" Oct 30 01:34:14.071967 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:14.071930 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:14.077943 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:14.077917 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:14.655270 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:14.655236 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:14.656125 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:14.656104 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:16.329948 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.329913 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh"] Oct 30 01:34:16.336076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.336052 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.357481 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.357311 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4d38e5-serving-cert\"" Oct 30 01:34:16.368051 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.368025 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh"] Oct 30 01:34:16.424344 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.424310 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59vzg\" (UniqueName: \"kubernetes.io/projected/923d6f91-f6f9-4881-9384-f7baf576bcb6-kube-api-access-59vzg\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.424547 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.424523 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-metrics-certs\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.424615 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.424592 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/923d6f91-f6f9-4881-9384-f7baf576bcb6-openshift-service-ca\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.424746 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.424733 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-maas-api-tls\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.525859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.525822 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-metrics-certs\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.526043 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.525908 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/923d6f91-f6f9-4881-9384-f7baf576bcb6-openshift-service-ca\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.526043 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.526029 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-maas-api-tls\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.526179 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.526063 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59vzg\" (UniqueName: \"kubernetes.io/projected/923d6f91-f6f9-4881-9384-f7baf576bcb6-kube-api-access-59vzg\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.526795 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.526760 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/923d6f91-f6f9-4881-9384-f7baf576bcb6-openshift-service-ca\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.530517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.529891 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-maas-api-tls\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.531276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.531247 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-metrics-certs\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.546925 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.546891 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59vzg\" (UniqueName: \"kubernetes.io/projected/923d6f91-f6f9-4881-9384-f7baf576bcb6-kube-api-access-59vzg\") pod \"maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.664160 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.664034 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:16.979474 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:16.979446 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh"] Oct 30 01:34:16.980595 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:34:16.980567 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod923d6f91_f6f9_4881_9384_f7baf576bcb6.slice/crio-e2fd2c94df78c84971b37710f4200f6989429d574ccef6fca329c755f080127b WatchSource:0}: Error finding container e2fd2c94df78c84971b37710f4200f6989429d574ccef6fca329c755f080127b: Status 404 returned error can't find the container with id e2fd2c94df78c84971b37710f4200f6989429d574ccef6fca329c755f080127b Oct 30 01:34:17.688484 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:17.687141 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" event={"ID":"923d6f91-f6f9-4881-9384-f7baf576bcb6","Type":"ContainerStarted","Data":"8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968"} Oct 30 01:34:17.688484 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:17.687206 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" event={"ID":"923d6f91-f6f9-4881-9384-f7baf576bcb6","Type":"ContainerStarted","Data":"e2fd2c94df78c84971b37710f4200f6989429d574ccef6fca329c755f080127b"} Oct 30 01:34:17.688484 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:17.688208 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:17.777096 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:17.776101 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" podStartSLOduration=1.3666233239999999 podStartE2EDuration="1.776080883s" totalImagesPullingTime="409.457559ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:17.773118067 +0000 UTC m=+1025.094075118" watchObservedRunningTime="2026-10-30 01:34:17.776080883 +0000 UTC m=+1025.097037921" Oct 30 01:34:18.367957 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.367926 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph"] Oct 30 01:34:18.372372 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.372351 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.412256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.410814 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph"] Oct 30 01:34:18.412256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.410926 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.412256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.410962 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.412256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.411209 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.412256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.411279 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.412256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.411306 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntsf\" (UniqueName: \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-kube-api-access-zntsf\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.412256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.411378 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76261f8-01b6-4764-be6b-cd36da5356ad-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.412256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.411413 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.412256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.411450 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.412256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.411480 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.512920 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.512888 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76261f8-01b6-4764-be6b-cd36da5356ad-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513105 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.512927 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513105 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.512943 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513105 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513051 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513286 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513145 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513286 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513171 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513452 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513430 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513497 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513511 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513680 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513633 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513680 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513644 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513695 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntsf\" (UniqueName: \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-kube-api-access-zntsf\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513866 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.513990 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.513966 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76261f8-01b6-4764-be6b-cd36da5356ad-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.515635 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.515617 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.515791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.515769 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.532891 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.532874 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.533268 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.533249 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntsf\" (UniqueName: \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-kube-api-access-zntsf\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:18.692274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:18.692195 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:19.013623 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:34:19.013583 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice/crio-384f19ba9e2a6dc40fa1c12de40ee44ee25a73cf91af45844e391461fe11d59c WatchSource:0}: Error finding container 384f19ba9e2a6dc40fa1c12de40ee44ee25a73cf91af45844e391461fe11d59c: Status 404 returned error can't find the container with id 384f19ba9e2a6dc40fa1c12de40ee44ee25a73cf91af45844e391461fe11d59c Oct 30 01:34:19.015771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:19.015743 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph"] Oct 30 01:34:19.016527 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:19.016494 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:19.016637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:19.016568 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:19.016637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:19.016612 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:19.227346 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:34:19.223569 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:34:19.704394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:19.704360 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" event={"ID":"a76261f8-01b6-4764-be6b-cd36da5356ad","Type":"ContainerStarted","Data":"364d65fc17f9484354bc685744f3deb975419bdad10a1607eae1d5fff4b89192"} Oct 30 01:34:19.704789 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:19.704446 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" event={"ID":"a76261f8-01b6-4764-be6b-cd36da5356ad","Type":"ContainerStarted","Data":"384f19ba9e2a6dc40fa1c12de40ee44ee25a73cf91af45844e391461fe11d59c"} Oct 30 01:34:19.791420 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:19.791359 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" podStartSLOduration=1.791338627 podStartE2EDuration="1.791338627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:19.759503269 +0000 UTC m=+1027.080460307" watchObservedRunningTime="2026-10-30 01:34:19.791338627 +0000 UTC m=+1027.112295657" Oct 30 01:34:20.693350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:20.692462 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:20.694606 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:20.694570 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:34:20.694756 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:20.694621 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:34:21.186392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.183865 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf"] Oct 30 01:34:21.189868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.189837 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.198042 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.196905 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf"] Oct 30 01:34:21.210366 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.206421 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-278fe7d5-openshift-default-dockercfg-gstz5\"" Oct 30 01:34:21.285350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.285203 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-envoy\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.285350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.285270 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-credential-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.285642 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.285448 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5743d668-2053-4276-a420-fd8a77559a01-istio-podinfo\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.285642 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.285498 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5743d668-2053-4276-a420-fd8a77559a01-istiod-ca-cert\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.285642 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.285592 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-certs\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.285806 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.285649 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-data\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.285806 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.285717 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4sm5\" (UniqueName: \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-kube-api-access-v4sm5\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.285806 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.285763 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-istio-token\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.285806 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.285795 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.398352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.397846 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4sm5\" (UniqueName: \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-kube-api-access-v4sm5\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.398352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.397911 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-istio-token\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.398352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.397944 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.398352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.398044 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-envoy\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.398352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.398083 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-credential-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.398352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.398143 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5743d668-2053-4276-a420-fd8a77559a01-istio-podinfo\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.398352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.398200 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5743d668-2053-4276-a420-fd8a77559a01-istiod-ca-cert\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.398352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.398285 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-certs\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.398901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.398448 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-data\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.398960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.398910 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-data\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.400230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.399776 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-credential-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.400230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.400099 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-certs\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.400515 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.400350 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5743d668-2053-4276-a420-fd8a77559a01-istiod-ca-cert\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.401037 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.400752 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-socket\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.403632 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.403597 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5743d668-2053-4276-a420-fd8a77559a01-istio-podinfo\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.404046 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.404019 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-envoy\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.418479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.418457 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-istio-token\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.419187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.419165 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4sm5\" (UniqueName: \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-kube-api-access-v4sm5\") pod \"e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.507621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.507589 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:21.693360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.692734 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:34:21.693360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.692798 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:34:21.907439 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.906904 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf"] Oct 30 01:34:21.911782 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.911214 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:21.911782 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.911299 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:21.911782 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:21.911362 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:22.694891 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:22.694798 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:34:22.695466 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:22.695420 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:34:22.731668 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:22.731636 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" event={"ID":"5743d668-2053-4276-a420-fd8a77559a01","Type":"ContainerStarted","Data":"195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae"} Oct 30 01:34:22.731840 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:22.731698 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" event={"ID":"5743d668-2053-4276-a420-fd8a77559a01","Type":"ContainerStarted","Data":"d4cb4137e50d7fe8c9005797e83a5a07d44eec552562156c834370ca6b95ce3b"} Oct 30 01:34:22.799045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:22.798983 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" podStartSLOduration=1.798963365 podStartE2EDuration="1.798963365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:22.798666459 +0000 UTC m=+1030.119623498" watchObservedRunningTime="2026-10-30 01:34:22.798963365 +0000 UTC m=+1030.119920406" Oct 30 01:34:23.512622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:23.509429 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:23.525457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:23.514432 5055 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:34:23.525457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:23.514510 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 01:34:23.698356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:23.697846 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:34:23.698356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:23.697923 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:34:24.516418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:24.515661 5055 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:34:24.516418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:24.515735 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 01:34:24.695352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:24.694744 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:34:24.695352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:24.694807 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:34:24.724360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:24.724249 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:34:25.508803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:25.508764 5055 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:34:25.508991 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:25.508816 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 01:34:25.694351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:25.693700 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 01:34:25.694351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:25.693760 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 01:34:26.510135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:26.510099 5055 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:34:26.510681 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:26.510150 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 01:34:27.509367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:27.509310 5055 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:34:27.509670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:27.509639 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 01:34:27.734586 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:27.734471 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:27.734586 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:27.734540 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:28.045685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.045604 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7495-d53493"] Oct 30 01:34:28.051587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.051554 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-d53493" Oct 30 01:34:28.058709 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.058676 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7495-d53493"] Oct 30 01:34:28.080352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.080198 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pm9kh\"" Oct 30 01:34:28.145724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.145690 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9jqb\" (UniqueName: \"kubernetes.io/projected/7dcad0fc-075a-497b-abe6-5b056bde1bb2-kube-api-access-z9jqb\") pod \"test-curl-7495-d53493\" (UID: \"7dcad0fc-075a-497b-abe6-5b056bde1bb2\") " pod="opendatahub/test-curl-7495-d53493" Oct 30 01:34:28.155018 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.154994 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:28.155463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.155450 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:28.156091 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.156065 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:34:28.247557 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.247525 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9jqb\" (UniqueName: \"kubernetes.io/projected/7dcad0fc-075a-497b-abe6-5b056bde1bb2-kube-api-access-z9jqb\") pod \"test-curl-7495-d53493\" (UID: \"7dcad0fc-075a-497b-abe6-5b056bde1bb2\") " pod="opendatahub/test-curl-7495-d53493" Oct 30 01:34:28.310413 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.309192 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9jqb\" (UniqueName: \"kubernetes.io/projected/7dcad0fc-075a-497b-abe6-5b056bde1bb2-kube-api-access-z9jqb\") pod \"test-curl-7495-d53493\" (UID: \"7dcad0fc-075a-497b-abe6-5b056bde1bb2\") " pod="opendatahub/test-curl-7495-d53493" Oct 30 01:34:28.444913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.444647 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-d53493" Oct 30 01:34:28.508818 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.508290 5055 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 01:34:28.508818 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.508428 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 01:34:28.773153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:28.773127 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7495-d53493"] Oct 30 01:34:29.512819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:29.512791 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:29.513305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:29.513285 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:29.513984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:29.513970 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:34:29.792351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:29.792247 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7495-d53493" event={"ID":"7dcad0fc-075a-497b-abe6-5b056bde1bb2","Type":"ContainerStarted","Data":"c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768"} Oct 30 01:34:29.792351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:29.792294 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7495-d53493" event={"ID":"7dcad0fc-075a-497b-abe6-5b056bde1bb2","Type":"ContainerStarted","Data":"a760cb9db8d9aebb01f5adb37925df7e21b83b8b85f57617eac33c5c82f6af68"} Oct 30 01:34:29.833031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:29.832969 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7495-d53493" podStartSLOduration=1.9946448650000002 podStartE2EDuration="2.832949634s" totalImagesPullingTime="838.304769ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:29.829756207 +0000 UTC m=+1037.150713248" watchObservedRunningTime="2026-10-30 01:34:29.832949634 +0000 UTC m=+1037.153906686" Oct 30 01:34:30.681348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:30.681303 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7495-d53493"] Oct 30 01:34:30.686655 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:30.686609 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7495-d53493"] Oct 30 01:34:31.806950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:31.806914 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7495-d53493" podUID="7dcad0fc-075a-497b-abe6-5b056bde1bb2" containerName="test-curl-7495-d53493" containerID="cri-o://c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768" gracePeriod=2 Oct 30 01:34:33.275342 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.275292 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq"] Oct 30 01:34:33.275896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.275882 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7dcad0fc-075a-497b-abe6-5b056bde1bb2" containerName="test-curl-7495-d53493" Oct 30 01:34:33.275951 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.275897 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcad0fc-075a-497b-abe6-5b056bde1bb2" containerName="test-curl-7495-d53493" Oct 30 01:34:33.276068 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.276057 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcad0fc-075a-497b-abe6-5b056bde1bb2" containerName="test-curl-7495-d53493" Oct 30 01:34:33.279817 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.279799 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.305912 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.305881 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a06822-serving-cert\"" Oct 30 01:34:33.312896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.312870 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq"] Oct 30 01:34:33.323356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.321896 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4980029-d407-4e49-9394-7f8feb460d35-openshift-service-ca\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.323356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.322091 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-metrics-certs\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.323356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.322167 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlvqf\" (UniqueName: \"kubernetes.io/projected/f4980029-d407-4e49-9394-7f8feb460d35-kube-api-access-dlvqf\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.323356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.322246 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-maas-api-tls\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.423218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.423186 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-metrics-certs\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.423401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.423245 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlvqf\" (UniqueName: \"kubernetes.io/projected/f4980029-d407-4e49-9394-7f8feb460d35-kube-api-access-dlvqf\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.423401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.423294 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-maas-api-tls\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.423526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.423455 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4980029-d407-4e49-9394-7f8feb460d35-openshift-service-ca\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.423526 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:34:33.423487 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-serving-cert: secret "maas-api-e2e-api-a-a06822-serving-cert" not found Oct 30 01:34:33.423624 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:34:33.423551 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-maas-api-tls podName:f4980029-d407-4e49-9394-7f8feb460d35 nodeName:}" failed. No retries permitted until 2026-10-30 01:34:33.92353207 +0000 UTC m=+1041.244489105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-maas-api-tls") pod "maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" (UID: "f4980029-d407-4e49-9394-7f8feb460d35") : secret "maas-api-e2e-api-a-a06822-serving-cert" not found Oct 30 01:34:33.424485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.424461 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4980029-d407-4e49-9394-7f8feb460d35-openshift-service-ca\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.426191 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.426169 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-metrics-certs\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.443718 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.443692 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlvqf\" (UniqueName: \"kubernetes.io/projected/f4980029-d407-4e49-9394-7f8feb460d35-kube-api-access-dlvqf\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.930966 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.930935 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-maas-api-tls\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:33.933524 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:33.933492 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-maas-api-tls\") pod \"maas-api-e2e-api-a-a06822-b8487ff66-sjlbq\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:34.032676 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.032652 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-d53493" Oct 30 01:34:34.133659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.133557 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcad0fc-075a-497b-abe6-5b056bde1bb2-kube-api-access-z9jqb\" (UniqueName: \"kubernetes.io/projected/7dcad0fc-075a-497b-abe6-5b056bde1bb2-kube-api-access-z9jqb\") pod \"7dcad0fc-075a-497b-abe6-5b056bde1bb2\" (UID: \"7dcad0fc-075a-497b-abe6-5b056bde1bb2\") " Oct 30 01:34:34.137388 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.137302 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcad0fc-075a-497b-abe6-5b056bde1bb2-kube-api-access-z9jqb" pod "7dcad0fc-075a-497b-abe6-5b056bde1bb2" (UID: "7dcad0fc-075a-497b-abe6-5b056bde1bb2"). InnerVolumeSpecName "kube-api-access-z9jqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:34:34.195647 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.195613 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:34.236070 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.236032 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9jqb\" (UniqueName: \"kubernetes.io/projected/7dcad0fc-075a-497b-abe6-5b056bde1bb2-kube-api-access-z9jqb\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:34:34.514022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.513989 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq"] Oct 30 01:34:34.829931 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.829869 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dcad0fc-075a-497b-abe6-5b056bde1bb2" path="/var/lib/kubelet/pods/7dcad0fc-075a-497b-abe6-5b056bde1bb2/volumes" Oct 30 01:34:34.849437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.849403 5055 generic.go:356] "Generic (PLEG): container finished" podID="7dcad0fc-075a-497b-abe6-5b056bde1bb2" containerID="c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768" exitCode=137 Oct 30 01:34:34.849591 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.849482 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-d53493" Oct 30 01:34:34.849591 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.849502 5055 scope.go:122] "RemoveContainer" containerID="c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768" Oct 30 01:34:34.854783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.854758 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" event={"ID":"f4980029-d407-4e49-9394-7f8feb460d35","Type":"ContainerStarted","Data":"8cb4ad95e36d4587bfcf8a311c243311c3a63ecceece8f3d6e898c7604fb8b0a"} Oct 30 01:34:34.872143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.872118 5055 scope.go:122] "RemoveContainer" containerID="c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768" Oct 30 01:34:34.872641 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:34:34.872619 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768\": container with ID starting with c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768 not found: ID does not exist" containerID="c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768" Oct 30 01:34:34.872720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:34.872653 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768"} err="failed to get container status \"c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768\": rpc error: code = NotFound desc = could not find container \"c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768\": container with ID starting with c7d971b55372992fbb67929994f467d5e3799c40812d4de8d48b1ede1eb85768 not found: ID does not exist" Oct 30 01:34:35.868971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:35.868935 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" event={"ID":"f4980029-d407-4e49-9394-7f8feb460d35","Type":"ContainerStarted","Data":"9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486"} Oct 30 01:34:35.869478 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:35.869022 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:35.915933 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:35.915779 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" podStartSLOduration=2.647992073 podStartE2EDuration="2.915762322s" totalImagesPullingTime="267.770249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:35.910431233 +0000 UTC m=+1043.231388281" watchObservedRunningTime="2026-10-30 01:34:35.915762322 +0000 UTC m=+1043.236719363" Oct 30 01:34:41.886120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:41.886091 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:34:49.222848 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:34:49.222810 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:34:55.086548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:55.086512 5055 generic.go:356] "Generic (PLEG): container finished" podID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerID="8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab" exitCode=0 Oct 30 01:34:55.086965 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:55.086594 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" event={"ID":"86c97571-2fe7-4f32-bade-7a725d5fce15","Type":"ContainerDied","Data":"8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab"} Oct 30 01:34:55.087101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:55.087084 5055 scope.go:122] "RemoveContainer" containerID="8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab" Oct 30 01:34:55.088689 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:55.088662 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:55.088804 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:55.088725 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:55.088804 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:55.088756 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:34:56.099571 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:56.099526 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" event={"ID":"86c97571-2fe7-4f32-bade-7a725d5fce15","Type":"ContainerStarted","Data":"0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7"} Oct 30 01:34:57.074353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:57.073653 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:34:57.089351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:57.082605 5055 patch_prober.go:29] interesting pod/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:34:57.089351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:57.082666 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:34:58.216076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:58.214812 5055 patch_prober.go:29] interesting pod/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:34:58.216076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:58.214871 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:34:59.117351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:59.106487 5055 patch_prober.go:29] interesting pod/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:34:59.117351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:59.106549 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:34:59.656266 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:34:59.656233 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:35:00.077357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:00.076485 5055 patch_prober.go:29] interesting pod/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:35:00.077357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:00.076545 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:35:01.044354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:01.041531 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5"] Oct 30 01:35:01.054348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:01.053721 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5012fc8d-pz5v5"] Oct 30 01:35:01.081408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:01.079731 5055 patch_prober.go:29] interesting pod/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 30 01:35:01.081408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:01.079807 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 30 01:35:02.076667 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:02.076637 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:35:02.077801 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:02.077782 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:35:02.822664 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:02.822628 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c170e56-00cc-4a35-a643-df4d790629b6" path="/var/lib/kubelet/pods/7c170e56-00cc-4a35-a643-df4d790629b6/volumes" Oct 30 01:35:12.033207 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:12.033172 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb"] Oct 30 01:35:12.043515 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:12.043490 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d5c792-4d2xb"] Oct 30 01:35:12.829358 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:12.829283 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27218363-120b-4c29-9656-96e9a2d96d89" path="/var/lib/kubelet/pods/27218363-120b-4c29-9656-96e9a2d96d89/volumes" Oct 30 01:35:14.831997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:14.831962 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds"] Oct 30 01:35:14.832499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:14.832176 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" podUID="43cb60a1-5ab7-45b9-b244-13600d3bee3c" containerName="maas-api" containerID="cri-o://6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15" gracePeriod=30 Oct 30 01:35:15.427592 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.427562 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:35:15.497130 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.497101 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43cb60a1-5ab7-45b9-b244-13600d3bee3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb60a1-5ab7-45b9-b244-13600d3bee3c-openshift-service-ca\") pod \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " Oct 30 01:35:15.498550 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.498521 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cb60a1-5ab7-45b9-b244-13600d3bee3c-openshift-service-ca" pod "43cb60a1-5ab7-45b9-b244-13600d3bee3c" (UID: "43cb60a1-5ab7-45b9-b244-13600d3bee3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:15.537672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.537633 5055 generic.go:356] "Generic (PLEG): container finished" podID="43cb60a1-5ab7-45b9-b244-13600d3bee3c" containerID="6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15" exitCode=0 Oct 30 01:35:15.537944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.537923 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" event={"ID":"43cb60a1-5ab7-45b9-b244-13600d3bee3c","Type":"ContainerDied","Data":"6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15"} Oct 30 01:35:15.538083 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.538068 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" event={"ID":"43cb60a1-5ab7-45b9-b244-13600d3bee3c","Type":"ContainerDied","Data":"c3af45edc50021bcbf73068f632a0b155bba52e71d11d1fe66d448a1b04c9425"} Oct 30 01:35:15.538199 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.538185 5055 scope.go:122] "RemoveContainer" containerID="6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15" Oct 30 01:35:15.538484 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.538465 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds" Oct 30 01:35:15.554464 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.554440 5055 scope.go:122] "RemoveContainer" containerID="6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15" Oct 30 01:35:15.554815 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:35:15.554796 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15\": container with ID starting with 6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15 not found: ID does not exist" containerID="6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15" Oct 30 01:35:15.554894 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.554836 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15"} err="failed to get container status \"6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15\": rpc error: code = NotFound desc = could not find container \"6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15\": container with ID starting with 6277fdf8cdc075978dc9bbb093de3d368688416f921b2d6af806629eaff18b15 not found: ID does not exist" Oct 30 01:35:15.601058 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.601019 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-metrics-certs\") pod \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " Oct 30 01:35:15.601348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.601310 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cb60a1-5ab7-45b9-b244-13600d3bee3c-kube-api-access-kcbqt\" (UniqueName: \"kubernetes.io/projected/43cb60a1-5ab7-45b9-b244-13600d3bee3c-kube-api-access-kcbqt\") pod \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " Oct 30 01:35:15.601770 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.601747 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-maas-api-tls\") pod \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\" (UID: \"43cb60a1-5ab7-45b9-b244-13600d3bee3c\") " Oct 30 01:35:15.602664 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.602642 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43cb60a1-5ab7-45b9-b244-13600d3bee3c-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:35:15.626274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.626238 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cb60a1-5ab7-45b9-b244-13600d3bee3c-kube-api-access-kcbqt" pod "43cb60a1-5ab7-45b9-b244-13600d3bee3c" (UID: "43cb60a1-5ab7-45b9-b244-13600d3bee3c"). InnerVolumeSpecName "kube-api-access-kcbqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:15.626706 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.626673 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-metrics-certs" pod "43cb60a1-5ab7-45b9-b244-13600d3bee3c" (UID: "43cb60a1-5ab7-45b9-b244-13600d3bee3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:15.627077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.627053 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-maas-api-tls" pod "43cb60a1-5ab7-45b9-b244-13600d3bee3c" (UID: "43cb60a1-5ab7-45b9-b244-13600d3bee3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:15.704397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.704361 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:35:15.704397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.704397 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43cb60a1-5ab7-45b9-b244-13600d3bee3c-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:35:15.704664 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.704410 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcbqt\" (UniqueName: \"kubernetes.io/projected/43cb60a1-5ab7-45b9-b244-13600d3bee3c-kube-api-access-kcbqt\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:35:15.892869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.892774 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds"] Oct 30 01:35:15.906802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:15.906772 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ec2345-5b7cf5b9fc-kjdds"] Oct 30 01:35:16.831174 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:16.831140 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43cb60a1-5ab7-45b9-b244-13600d3bee3c" path="/var/lib/kubelet/pods/43cb60a1-5ab7-45b9-b244-13600d3bee3c/volumes" Oct 30 01:35:17.888639 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.888602 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg"] Oct 30 01:35:17.889385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.889368 5055 cpu_manager.go:414] "containerMap: removing container" podUID="43cb60a1-5ab7-45b9-b244-13600d3bee3c" containerName="maas-api" Oct 30 01:35:17.889449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.889386 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cb60a1-5ab7-45b9-b244-13600d3bee3c" containerName="maas-api" Oct 30 01:35:17.889493 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.889483 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cb60a1-5ab7-45b9-b244-13600d3bee3c" containerName="maas-api" Oct 30 01:35:17.894460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.894441 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:17.910870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.910835 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-97c137-serving-cert\"" Oct 30 01:35:17.930062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.930028 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg"] Oct 30 01:35:17.988802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.988763 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpwkb\" (UniqueName: \"kubernetes.io/projected/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-kube-api-access-gpwkb\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:17.988983 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.988935 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-maas-api-tls\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:17.989076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.989010 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-metrics-certs\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:17.989249 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:17.989223 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-openshift-service-ca\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:18.101350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.097758 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-openshift-service-ca\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:18.101350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.097851 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpwkb\" (UniqueName: \"kubernetes.io/projected/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-kube-api-access-gpwkb\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:18.101625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.101371 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-openshift-service-ca\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:18.116422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.116319 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-maas-api-tls\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:18.116629 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.116433 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-metrics-certs\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:18.123569 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.122425 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-maas-api-tls\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:18.125233 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.125205 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-metrics-certs\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:18.126153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.126128 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpwkb\" (UniqueName: \"kubernetes.io/projected/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-kube-api-access-gpwkb\") pod \"maas-api-e2e-api-b-97c137-6dc57d588-ppllg\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:18.209500 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.209473 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:18.517736 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.517707 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg"] Oct 30 01:35:18.518812 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:35:18.518785 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79ffe20_20bc_4e7d_9298_e7cb8d94d930.slice/crio-cc3dc24cdb31a756bc73b8cc34e8d8cd5a105bd9e9a62b5d2502dfad843eb4a9 WatchSource:0}: Error finding container cc3dc24cdb31a756bc73b8cc34e8d8cd5a105bd9e9a62b5d2502dfad843eb4a9: Status 404 returned error can't find the container with id cc3dc24cdb31a756bc73b8cc34e8d8cd5a105bd9e9a62b5d2502dfad843eb4a9 Oct 30 01:35:18.571428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:18.571395 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" event={"ID":"b79ffe20-20bc-4e7d-9298-e7cb8d94d930","Type":"ContainerStarted","Data":"cc3dc24cdb31a756bc73b8cc34e8d8cd5a105bd9e9a62b5d2502dfad843eb4a9"} Oct 30 01:35:19.029537 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:19.029493 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5"] Oct 30 01:35:19.042675 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:19.042638 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11891f4f-2nrx5"] Oct 30 01:35:19.229264 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:35:19.229227 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:35:19.585110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:19.584977 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" event={"ID":"b79ffe20-20bc-4e7d-9298-e7cb8d94d930","Type":"ContainerStarted","Data":"c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3"} Oct 30 01:35:19.585110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:19.585091 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:19.627837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:19.627760 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" podStartSLOduration=2.253240885 podStartE2EDuration="2.627738716s" totalImagesPullingTime="374.497831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:19.626174519 +0000 UTC m=+1086.947131560" watchObservedRunningTime="2026-10-30 01:35:19.627738716 +0000 UTC m=+1086.948695794" Oct 30 01:35:20.821564 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:20.821532 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f86095a6-b32e-49b2-98c5-3a2b5925d161" path="/var/lib/kubelet/pods/f86095a6-b32e-49b2-98c5-3a2b5925d161/volumes" Oct 30 01:35:21.549156 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.549123 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d"] Oct 30 01:35:21.553622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.553600 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.581146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.581119 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-288c8f-serving-cert\"" Oct 30 01:35:21.597883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.597840 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d"] Oct 30 01:35:21.684267 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.684222 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqfb\" (UniqueName: \"kubernetes.io/projected/0ad750c7-80da-4d24-af92-37b066980ff0-kube-api-access-pzqfb\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.684493 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.684460 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-maas-api-tls\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.684611 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.684589 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-metrics-certs\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.684750 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.684732 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad750c7-80da-4d24-af92-37b066980ff0-openshift-service-ca\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.786119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.786085 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad750c7-80da-4d24-af92-37b066980ff0-openshift-service-ca\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.786313 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.786141 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqfb\" (UniqueName: \"kubernetes.io/projected/0ad750c7-80da-4d24-af92-37b066980ff0-kube-api-access-pzqfb\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.786313 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.786219 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-maas-api-tls\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.786313 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.786291 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-metrics-certs\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.787001 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.786969 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad750c7-80da-4d24-af92-37b066980ff0-openshift-service-ca\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.789076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.789050 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-metrics-certs\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.789240 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.789218 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-maas-api-tls\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.806823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.806752 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqfb\" (UniqueName: \"kubernetes.io/projected/0ad750c7-80da-4d24-af92-37b066980ff0-kube-api-access-pzqfb\") pod \"maas-api-e2e-api-a-288c8f-7b876765dd-cq82d\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:21.869973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:21.869925 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:22.186952 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:35:22.186922 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad750c7_80da_4d24_af92_37b066980ff0.slice/crio-b7ab81ce56ef31d1db2907a01c4e00e45ddb568ef8c690539558f5ed732d81c5 WatchSource:0}: Error finding container b7ab81ce56ef31d1db2907a01c4e00e45ddb568ef8c690539558f5ed732d81c5: Status 404 returned error can't find the container with id b7ab81ce56ef31d1db2907a01c4e00e45ddb568ef8c690539558f5ed732d81c5 Oct 30 01:35:22.189880 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:22.189852 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d"] Oct 30 01:35:22.618514 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:22.618482 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" event={"ID":"0ad750c7-80da-4d24-af92-37b066980ff0","Type":"ContainerStarted","Data":"b7ab81ce56ef31d1db2907a01c4e00e45ddb568ef8c690539558f5ed732d81c5"} Oct 30 01:35:23.376720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.376539 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw"] Oct 30 01:35:23.389349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.382854 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.425348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.423075 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-278fe7d5-serving-cert\"" Oct 30 01:35:23.444565 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.437024 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw"] Oct 30 01:35:23.544348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.535735 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c0468c-b08a-4f98-87f2-407d678d039d-openshift-service-ca\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.544348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.535797 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-maas-api-tls\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.544348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.535905 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8g4\" (UniqueName: \"kubernetes.io/projected/62c0468c-b08a-4f98-87f2-407d678d039d-kube-api-access-dn8g4\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.544348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.536026 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-metrics-certs\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.663404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.658313 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-metrics-certs\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.663404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.658510 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c0468c-b08a-4f98-87f2-407d678d039d-openshift-service-ca\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.663404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.658551 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-maas-api-tls\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.663404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.658662 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8g4\" (UniqueName: \"kubernetes.io/projected/62c0468c-b08a-4f98-87f2-407d678d039d-kube-api-access-dn8g4\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.663404 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:35:23.659137 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-serving-cert: secret "maas-api-e2e-mt-278fe7d5-serving-cert" not found Oct 30 01:35:23.663404 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:35:23.659200 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-maas-api-tls podName:62c0468c-b08a-4f98-87f2-407d678d039d nodeName:}" failed. No retries permitted until 2026-10-30 01:35:24.159179293 +0000 UTC m=+1091.480136327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-maas-api-tls") pod "maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" (UID: "62c0468c-b08a-4f98-87f2-407d678d039d") : secret "maas-api-e2e-mt-278fe7d5-serving-cert" not found Oct 30 01:35:23.663404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.659275 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c0468c-b08a-4f98-87f2-407d678d039d-openshift-service-ca\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.727374 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.724186 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8g4\" (UniqueName: \"kubernetes.io/projected/62c0468c-b08a-4f98-87f2-407d678d039d-kube-api-access-dn8g4\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.753960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.753931 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-metrics-certs\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:23.770899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.770865 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" event={"ID":"0ad750c7-80da-4d24-af92-37b066980ff0","Type":"ContainerStarted","Data":"c7e31ed34912dc396623fde0221708db693f9d5b8f3611fc4eb26be8d7dad28a"} Oct 30 01:35:23.771377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.771353 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:23.811495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:23.811440 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" podStartSLOduration=2.459236372 podStartE2EDuration="2.811419738s" totalImagesPullingTime="352.183366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:23.810020258 +0000 UTC m=+1091.130977298" watchObservedRunningTime="2026-10-30 01:35:23.811419738 +0000 UTC m=+1091.132376779" Oct 30 01:35:24.171957 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:24.171915 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-maas-api-tls\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:24.174663 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:24.174635 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-maas-api-tls\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:24.437740 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:24.437653 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:24.863402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:24.863378 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw"] Oct 30 01:35:25.599004 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:25.598977 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:35:25.839489 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:25.839458 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" event={"ID":"62c0468c-b08a-4f98-87f2-407d678d039d","Type":"ContainerStarted","Data":"f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4"} Oct 30 01:35:25.839672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:25.839514 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" event={"ID":"62c0468c-b08a-4f98-87f2-407d678d039d","Type":"ContainerStarted","Data":"1e73f8d0a62b18155981449df46da539f657d0c88ee9ce8c2cf02f082672c4b0"} Oct 30 01:35:25.839672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:25.839560 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:25.881047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:25.880936 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" podStartSLOduration=2.545084535 podStartE2EDuration="2.880920869s" totalImagesPullingTime="335.836334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:25.877758232 +0000 UTC m=+1093.198715272" watchObservedRunningTime="2026-10-30 01:35:25.880920869 +0000 UTC m=+1093.201877908" Oct 30 01:35:29.837871 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:29.837841 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:35:31.851959 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:31.851934 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:35:48.526805 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:48.526764 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7495-c5270f"] Oct 30 01:35:48.532988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:48.532966 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-c5270f" Oct 30 01:35:48.548376 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:48.548293 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7495-c5270f"] Oct 30 01:35:48.552901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:48.552867 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt2jx\" (UniqueName: \"kubernetes.io/projected/520869fc-2de8-4d6b-8f8e-2a5b5960d638-kube-api-access-zt2jx\") pod \"test-curl-7495-c5270f\" (UID: \"520869fc-2de8-4d6b-8f8e-2a5b5960d638\") " pod="opendatahub/test-curl-7495-c5270f" Oct 30 01:35:48.555368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:48.555306 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pm9kh\"" Oct 30 01:35:48.654179 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:48.654137 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt2jx\" (UniqueName: \"kubernetes.io/projected/520869fc-2de8-4d6b-8f8e-2a5b5960d638-kube-api-access-zt2jx\") pod \"test-curl-7495-c5270f\" (UID: \"520869fc-2de8-4d6b-8f8e-2a5b5960d638\") " pod="opendatahub/test-curl-7495-c5270f" Oct 30 01:35:48.680280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:48.680247 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt2jx\" (UniqueName: \"kubernetes.io/projected/520869fc-2de8-4d6b-8f8e-2a5b5960d638-kube-api-access-zt2jx\") pod \"test-curl-7495-c5270f\" (UID: \"520869fc-2de8-4d6b-8f8e-2a5b5960d638\") " pod="opendatahub/test-curl-7495-c5270f" Oct 30 01:35:48.846706 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:48.846619 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-c5270f" Oct 30 01:35:49.146000 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:35:49.145968 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod520869fc_2de8_4d6b_8f8e_2a5b5960d638.slice/crio-45e0aceed854b6d060f5e491d6c4bb3e78604342a881de3765a1ff12182b1d05 WatchSource:0}: Error finding container 45e0aceed854b6d060f5e491d6c4bb3e78604342a881de3765a1ff12182b1d05: Status 404 returned error can't find the container with id 45e0aceed854b6d060f5e491d6c4bb3e78604342a881de3765a1ff12182b1d05 Oct 30 01:35:49.147094 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:49.147072 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7495-c5270f"] Oct 30 01:35:49.228364 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:35:49.228309 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:35:50.035967 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:50.035932 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7495-c5270f" event={"ID":"520869fc-2de8-4d6b-8f8e-2a5b5960d638","Type":"ContainerStarted","Data":"9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073"} Oct 30 01:35:50.036504 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:50.035977 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7495-c5270f" event={"ID":"520869fc-2de8-4d6b-8f8e-2a5b5960d638","Type":"ContainerStarted","Data":"45e0aceed854b6d060f5e491d6c4bb3e78604342a881de3765a1ff12182b1d05"} Oct 30 01:35:50.078625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:50.078568 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7495-c5270f" podStartSLOduration=2.078550823 podStartE2EDuration="2.078550823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:50.074688017 +0000 UTC m=+1117.395645071" watchObservedRunningTime="2026-10-30 01:35:50.078550823 +0000 UTC m=+1117.399507864" Oct 30 01:35:50.939039 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:50.939006 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7495-c5270f"] Oct 30 01:35:50.943343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:50.943297 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7495-c5270f"] Oct 30 01:35:52.051381 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:52.051347 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7495-c5270f" podUID="520869fc-2de8-4d6b-8f8e-2a5b5960d638" containerName="test-curl-7495-c5270f" containerID="cri-o://9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073" gracePeriod=2 Oct 30 01:35:52.066584 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:35:52.066548 5055 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7495-c5270f\" is forbidden: User \"system:node:ip-10-0-134-254.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-254.ec2.internal' and this object" podUID="520869fc-2de8-4d6b-8f8e-2a5b5960d638" pod="opendatahub/test-curl-7495-c5270f" Oct 30 01:35:52.831694 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:35:52.831660 5055 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7495-c5270f\" is forbidden: User \"system:node:ip-10-0-134-254.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-254.ec2.internal' and this object" podUID="520869fc-2de8-4d6b-8f8e-2a5b5960d638" pod="opendatahub/test-curl-7495-c5270f" Oct 30 01:35:54.310740 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:54.310679 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-c5270f" Oct 30 01:35:54.441465 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:54.441434 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520869fc-2de8-4d6b-8f8e-2a5b5960d638-kube-api-access-zt2jx\" (UniqueName: \"kubernetes.io/projected/520869fc-2de8-4d6b-8f8e-2a5b5960d638-kube-api-access-zt2jx\") pod \"520869fc-2de8-4d6b-8f8e-2a5b5960d638\" (UID: \"520869fc-2de8-4d6b-8f8e-2a5b5960d638\") " Oct 30 01:35:54.443757 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:54.443736 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520869fc-2de8-4d6b-8f8e-2a5b5960d638-kube-api-access-zt2jx" pod "520869fc-2de8-4d6b-8f8e-2a5b5960d638" (UID: "520869fc-2de8-4d6b-8f8e-2a5b5960d638"). InnerVolumeSpecName "kube-api-access-zt2jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:54.543432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:54.543401 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt2jx\" (UniqueName: \"kubernetes.io/projected/520869fc-2de8-4d6b-8f8e-2a5b5960d638-kube-api-access-zt2jx\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:35:54.824087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:54.824057 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520869fc-2de8-4d6b-8f8e-2a5b5960d638" path="/var/lib/kubelet/pods/520869fc-2de8-4d6b-8f8e-2a5b5960d638/volumes" Oct 30 01:35:55.077177 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:55.077078 5055 generic.go:356] "Generic (PLEG): container finished" podID="520869fc-2de8-4d6b-8f8e-2a5b5960d638" containerID="9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073" exitCode=137 Oct 30 01:35:55.077177 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:55.077130 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-c5270f" Oct 30 01:35:55.077177 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:55.077176 5055 scope.go:122] "RemoveContainer" containerID="9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073" Oct 30 01:35:55.092471 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:55.092448 5055 scope.go:122] "RemoveContainer" containerID="9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073" Oct 30 01:35:55.093175 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:35:55.093142 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073\": container with ID starting with 9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073 not found: ID does not exist" containerID="9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073" Oct 30 01:35:55.093313 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:35:55.093185 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073"} err="failed to get container status \"9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073\": rpc error: code = NotFound desc = could not find container \"9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073\": container with ID starting with 9fd4bf3c8f1a5eae908071dbde8be7435acf1d7b34050c9885bb8397c52fb073 not found: ID does not exist" Oct 30 01:36:06.057455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.057418 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh"] Oct 30 01:36:06.057955 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.057680 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" podUID="923d6f91-f6f9-4881-9384-f7baf576bcb6" containerName="maas-api" containerID="cri-o://8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968" gracePeriod=30 Oct 30 01:36:06.572704 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.572675 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:36:06.713406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.699885 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-maas-api-tls\") pod \"923d6f91-f6f9-4881-9384-f7baf576bcb6\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " Oct 30 01:36:06.730317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.718470 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/923d6f91-f6f9-4881-9384-f7baf576bcb6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/923d6f91-f6f9-4881-9384-f7baf576bcb6-openshift-service-ca\") pod \"923d6f91-f6f9-4881-9384-f7baf576bcb6\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " Oct 30 01:36:06.730317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.725602 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/923d6f91-f6f9-4881-9384-f7baf576bcb6-openshift-service-ca" pod "923d6f91-f6f9-4881-9384-f7baf576bcb6" (UID: "923d6f91-f6f9-4881-9384-f7baf576bcb6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:06.730317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.725770 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/923d6f91-f6f9-4881-9384-f7baf576bcb6-kube-api-access-59vzg\" (UniqueName: \"kubernetes.io/projected/923d6f91-f6f9-4881-9384-f7baf576bcb6-kube-api-access-59vzg\") pod \"923d6f91-f6f9-4881-9384-f7baf576bcb6\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " Oct 30 01:36:06.730317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.726269 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-metrics-certs\") pod \"923d6f91-f6f9-4881-9384-f7baf576bcb6\" (UID: \"923d6f91-f6f9-4881-9384-f7baf576bcb6\") " Oct 30 01:36:06.730317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.727437 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/923d6f91-f6f9-4881-9384-f7baf576bcb6-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:06.739932 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.739838 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-metrics-certs" pod "923d6f91-f6f9-4881-9384-f7baf576bcb6" (UID: "923d6f91-f6f9-4881-9384-f7baf576bcb6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:06.740952 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.740928 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-maas-api-tls" pod "923d6f91-f6f9-4881-9384-f7baf576bcb6" (UID: "923d6f91-f6f9-4881-9384-f7baf576bcb6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:06.742817 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.742779 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/923d6f91-f6f9-4881-9384-f7baf576bcb6-kube-api-access-59vzg" pod "923d6f91-f6f9-4881-9384-f7baf576bcb6" (UID: "923d6f91-f6f9-4881-9384-f7baf576bcb6"). InnerVolumeSpecName "kube-api-access-59vzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:06.830639 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.830605 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:06.830639 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.830638 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59vzg\" (UniqueName: \"kubernetes.io/projected/923d6f91-f6f9-4881-9384-f7baf576bcb6-kube-api-access-59vzg\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:06.830887 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:06.830649 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/923d6f91-f6f9-4881-9384-f7baf576bcb6-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:07.178910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:07.178871 5055 generic.go:356] "Generic (PLEG): container finished" podID="923d6f91-f6f9-4881-9384-f7baf576bcb6" containerID="8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968" exitCode=0 Oct 30 01:36:07.179415 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:07.178927 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" Oct 30 01:36:07.179415 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:07.178945 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" event={"ID":"923d6f91-f6f9-4881-9384-f7baf576bcb6","Type":"ContainerDied","Data":"8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968"} Oct 30 01:36:07.179415 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:07.178987 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh" event={"ID":"923d6f91-f6f9-4881-9384-f7baf576bcb6","Type":"ContainerDied","Data":"e2fd2c94df78c84971b37710f4200f6989429d574ccef6fca329c755f080127b"} Oct 30 01:36:07.179415 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:07.179015 5055 scope.go:122] "RemoveContainer" containerID="8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968" Oct 30 01:36:07.192339 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:07.192303 5055 scope.go:122] "RemoveContainer" containerID="8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968" Oct 30 01:36:07.192664 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:36:07.192640 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968\": container with ID starting with 8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968 not found: ID does not exist" containerID="8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968" Oct 30 01:36:07.192734 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:07.192677 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968"} err="failed to get container status \"8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968\": rpc error: code = NotFound desc = could not find container \"8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968\": container with ID starting with 8eb8b3e95e6149ee8126bb226e0a5e15623503aabb22c83ef42bc9a0a51fa968 not found: ID does not exist" Oct 30 01:36:07.231487 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:07.231463 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh"] Oct 30 01:36:07.243622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:07.243599 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4d38e5-65889d9cf5-s8dwh"] Oct 30 01:36:08.822526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:08.822496 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="923d6f91-f6f9-4881-9384-f7baf576bcb6" path="/var/lib/kubelet/pods/923d6f91-f6f9-4881-9384-f7baf576bcb6/volumes" Oct 30 01:36:13.267753 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:13.267719 5055 scope.go:122] "RemoveContainer" containerID="a21ae8acd82c5543efb36855c9569e427731d2f1774846d9536a6af0f7430975" Oct 30 01:36:13.278583 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:13.278555 5055 scope.go:122] "RemoveContainer" containerID="74aa770af2ea411335750841d5f1742c67fae29dcaf0d47c38224bf682bb0656" Oct 30 01:36:18.492899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.492867 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw"] Oct 30 01:36:18.493391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.493122 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" podUID="62c0468c-b08a-4f98-87f2-407d678d039d" containerName="maas-api" containerID="cri-o://f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4" gracePeriod=30 Oct 30 01:36:18.823262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.823232 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:36:18.922576 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.921891 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c0468c-b08a-4f98-87f2-407d678d039d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c0468c-b08a-4f98-87f2-407d678d039d-openshift-service-ca\") pod \"62c0468c-b08a-4f98-87f2-407d678d039d\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " Oct 30 01:36:18.922576 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.921954 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c0468c-b08a-4f98-87f2-407d678d039d-kube-api-access-dn8g4\" (UniqueName: \"kubernetes.io/projected/62c0468c-b08a-4f98-87f2-407d678d039d-kube-api-access-dn8g4\") pod \"62c0468c-b08a-4f98-87f2-407d678d039d\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " Oct 30 01:36:18.922576 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.921993 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-maas-api-tls\") pod \"62c0468c-b08a-4f98-87f2-407d678d039d\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " Oct 30 01:36:18.922576 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.922277 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-metrics-certs\") pod \"62c0468c-b08a-4f98-87f2-407d678d039d\" (UID: \"62c0468c-b08a-4f98-87f2-407d678d039d\") " Oct 30 01:36:18.925853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.923736 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c0468c-b08a-4f98-87f2-407d678d039d-openshift-service-ca" pod "62c0468c-b08a-4f98-87f2-407d678d039d" (UID: "62c0468c-b08a-4f98-87f2-407d678d039d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:18.936461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.930420 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c0468c-b08a-4f98-87f2-407d678d039d-kube-api-access-dn8g4" pod "62c0468c-b08a-4f98-87f2-407d678d039d" (UID: "62c0468c-b08a-4f98-87f2-407d678d039d"). InnerVolumeSpecName "kube-api-access-dn8g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:18.936461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.930492 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-maas-api-tls" pod "62c0468c-b08a-4f98-87f2-407d678d039d" (UID: "62c0468c-b08a-4f98-87f2-407d678d039d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:18.936461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:18.931764 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-metrics-certs" pod "62c0468c-b08a-4f98-87f2-407d678d039d" (UID: "62c0468c-b08a-4f98-87f2-407d678d039d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:19.031546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.024735 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.031546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.024775 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c0468c-b08a-4f98-87f2-407d678d039d-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.031546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.024790 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn8g4\" (UniqueName: \"kubernetes.io/projected/62c0468c-b08a-4f98-87f2-407d678d039d-kube-api-access-dn8g4\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.031546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.024805 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c0468c-b08a-4f98-87f2-407d678d039d-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.244210 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:36:19.244154 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:36:19.279442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.274800 5055 generic.go:356] "Generic (PLEG): container finished" podID="62c0468c-b08a-4f98-87f2-407d678d039d" containerID="f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4" exitCode=0 Oct 30 01:36:19.279442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.275028 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" event={"ID":"62c0468c-b08a-4f98-87f2-407d678d039d","Type":"ContainerDied","Data":"f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4"} Oct 30 01:36:19.279442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.275103 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" event={"ID":"62c0468c-b08a-4f98-87f2-407d678d039d","Type":"ContainerDied","Data":"1e73f8d0a62b18155981449df46da539f657d0c88ee9ce8c2cf02f082672c4b0"} Oct 30 01:36:19.279442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.275137 5055 scope.go:122] "RemoveContainer" containerID="f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4" Oct 30 01:36:19.279442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.275396 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw" Oct 30 01:36:19.307766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.307741 5055 scope.go:122] "RemoveContainer" containerID="f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4" Oct 30 01:36:19.308853 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:36:19.308815 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4\": container with ID starting with f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4 not found: ID does not exist" containerID="f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4" Oct 30 01:36:19.308973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.308863 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4"} err="failed to get container status \"f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4\": rpc error: code = NotFound desc = could not find container \"f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4\": container with ID starting with f8b9a008098923b6ac5d256be976a4177adc1a5ca3c71053a15eb8f2a36533d4 not found: ID does not exist" Oct 30 01:36:19.343160 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.343133 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw"] Oct 30 01:36:19.352836 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:19.352803 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-rd9dw"] Oct 30 01:36:20.821280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:20.821245 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c0468c-b08a-4f98-87f2-407d678d039d" path="/var/lib/kubelet/pods/62c0468c-b08a-4f98-87f2-407d678d039d/volumes" Oct 30 01:36:20.844808 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:20.844779 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq"] Oct 30 01:36:20.845002 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:20.844956 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" podUID="f4980029-d407-4e49-9394-7f8feb460d35" containerName="maas-api" containerID="cri-o://9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486" gracePeriod=30 Oct 30 01:36:21.267647 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.267614 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:36:21.316350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.311263 5055 generic.go:356] "Generic (PLEG): container finished" podID="f4980029-d407-4e49-9394-7f8feb460d35" containerID="9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486" exitCode=0 Oct 30 01:36:21.316350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.311531 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" event={"ID":"f4980029-d407-4e49-9394-7f8feb460d35","Type":"ContainerDied","Data":"9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486"} Oct 30 01:36:21.316350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.311587 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" event={"ID":"f4980029-d407-4e49-9394-7f8feb460d35","Type":"ContainerDied","Data":"8cb4ad95e36d4587bfcf8a311c243311c3a63ecceece8f3d6e898c7604fb8b0a"} Oct 30 01:36:21.316350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.311616 5055 scope.go:122] "RemoveContainer" containerID="9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486" Oct 30 01:36:21.316350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.311759 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq" Oct 30 01:36:21.356003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.355972 5055 scope.go:122] "RemoveContainer" containerID="9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486" Oct 30 01:36:21.356564 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:36:21.356532 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486\": container with ID starting with 9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486 not found: ID does not exist" containerID="9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486" Oct 30 01:36:21.356660 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.356574 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486"} err="failed to get container status \"9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486\": rpc error: code = NotFound desc = could not find container \"9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486\": container with ID starting with 9545b58d7f98a1fc2a9f993bd0c2a6957287abff3f22374965b1bb8874545486 not found: ID does not exist" Oct 30 01:36:21.399348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.398785 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4980029-d407-4e49-9394-7f8feb460d35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4980029-d407-4e49-9394-7f8feb460d35-openshift-service-ca\") pod \"f4980029-d407-4e49-9394-7f8feb460d35\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " Oct 30 01:36:21.399348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.398849 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-metrics-certs\") pod \"f4980029-d407-4e49-9394-7f8feb460d35\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " Oct 30 01:36:21.399348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.398917 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-maas-api-tls\") pod \"f4980029-d407-4e49-9394-7f8feb460d35\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " Oct 30 01:36:21.399348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.399139 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4980029-d407-4e49-9394-7f8feb460d35-kube-api-access-dlvqf\" (UniqueName: \"kubernetes.io/projected/f4980029-d407-4e49-9394-7f8feb460d35-kube-api-access-dlvqf\") pod \"f4980029-d407-4e49-9394-7f8feb460d35\" (UID: \"f4980029-d407-4e49-9394-7f8feb460d35\") " Oct 30 01:36:21.399348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.399245 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4980029-d407-4e49-9394-7f8feb460d35-openshift-service-ca" pod "f4980029-d407-4e49-9394-7f8feb460d35" (UID: "f4980029-d407-4e49-9394-7f8feb460d35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:21.402366 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.399794 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4980029-d407-4e49-9394-7f8feb460d35-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:21.404683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.404656 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4980029-d407-4e49-9394-7f8feb460d35-kube-api-access-dlvqf" pod "f4980029-d407-4e49-9394-7f8feb460d35" (UID: "f4980029-d407-4e49-9394-7f8feb460d35"). InnerVolumeSpecName "kube-api-access-dlvqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:21.406104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.406072 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-maas-api-tls" pod "f4980029-d407-4e49-9394-7f8feb460d35" (UID: "f4980029-d407-4e49-9394-7f8feb460d35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:21.408466 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.408448 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-metrics-certs" pod "f4980029-d407-4e49-9394-7f8feb460d35" (UID: "f4980029-d407-4e49-9394-7f8feb460d35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:21.509350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.506440 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:21.509350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.506469 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4980029-d407-4e49-9394-7f8feb460d35-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:21.509350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.506482 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlvqf\" (UniqueName: \"kubernetes.io/projected/f4980029-d407-4e49-9394-7f8feb460d35-kube-api-access-dlvqf\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:21.662767 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.662735 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq"] Oct 30 01:36:21.668192 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:21.668164 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a06822-b8487ff66-sjlbq"] Oct 30 01:36:22.822337 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:22.822304 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4980029-d407-4e49-9394-7f8feb460d35" path="/var/lib/kubelet/pods/f4980029-d407-4e49-9394-7f8feb460d35/volumes" Oct 30 01:36:28.557675 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:28.557640 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj"] Oct 30 01:36:28.558885 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:28.558854 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" containerID="cri-o://0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7" gracePeriod=30 Oct 30 01:36:29.520837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.520802 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq"] Oct 30 01:36:29.522028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.521994 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f4980029-d407-4e49-9394-7f8feb460d35" containerName="maas-api" Oct 30 01:36:29.522028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522026 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4980029-d407-4e49-9394-7f8feb460d35" containerName="maas-api" Oct 30 01:36:29.522239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522079 5055 cpu_manager.go:414] "containerMap: removing container" podUID="923d6f91-f6f9-4881-9384-f7baf576bcb6" containerName="maas-api" Oct 30 01:36:29.522239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522086 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="923d6f91-f6f9-4881-9384-f7baf576bcb6" containerName="maas-api" Oct 30 01:36:29.522239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522102 5055 cpu_manager.go:414] "containerMap: removing container" podUID="520869fc-2de8-4d6b-8f8e-2a5b5960d638" containerName="test-curl-7495-c5270f" Oct 30 01:36:29.522239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522110 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520869fc-2de8-4d6b-8f8e-2a5b5960d638" containerName="test-curl-7495-c5270f" Oct 30 01:36:29.522239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522140 5055 cpu_manager.go:414] "containerMap: removing container" podUID="62c0468c-b08a-4f98-87f2-407d678d039d" containerName="maas-api" Oct 30 01:36:29.522239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522147 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c0468c-b08a-4f98-87f2-407d678d039d" containerName="maas-api" Oct 30 01:36:29.522566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522348 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="923d6f91-f6f9-4881-9384-f7baf576bcb6" containerName="maas-api" Oct 30 01:36:29.522566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522362 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="520869fc-2de8-4d6b-8f8e-2a5b5960d638" containerName="test-curl-7495-c5270f" Oct 30 01:36:29.522566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522380 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c0468c-b08a-4f98-87f2-407d678d039d" containerName="maas-api" Oct 30 01:36:29.522566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.522395 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4980029-d407-4e49-9394-7f8feb460d35" containerName="maas-api" Oct 30 01:36:29.528480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.528458 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq"] Oct 30 01:36:29.528617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.528550 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.545038 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.545013 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f09934-openshift-default-dockercfg-w6jb7\"" Oct 30 01:36:29.655388 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.651306 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-certs\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.662619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.662571 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-socket\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.666555 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.666525 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-envoy\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.671460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.671419 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-podinfo\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.671759 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.671728 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-data\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.674574 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.674552 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-token\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.681542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.681497 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istiod-ca-cert\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.681736 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.681716 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g977\" (UniqueName: \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-kube-api-access-4g977\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.681918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.681889 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-credential-socket\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.787135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.787041 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-credential-socket\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.787897 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.787875 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-certs\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.788428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.788409 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-socket\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.788895 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.788858 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-envoy\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.789387 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.789370 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-podinfo\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.790028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.790010 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-data\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.790594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.790577 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-token\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.790825 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.790805 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istiod-ca-cert\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.790964 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.790936 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g977\" (UniqueName: \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-kube-api-access-4g977\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.808345 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.790438 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-data\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.808345 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.787715 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-credential-socket\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.808345 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.788286 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-certs\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.808345 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.788782 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-socket\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.808345 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.805392 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istiod-ca-cert\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.820206 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.819636 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-envoy\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.826347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.822423 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-podinfo\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.830351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.829857 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-token\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.854454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.843968 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g977\" (UniqueName: \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-kube-api-access-4g977\") pod \"e2e-api-a-f09934-openshift-default-748c86d47-27bdq\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:29.854454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:29.851941 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:30.234048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:30.233224 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq"] Oct 30 01:36:30.237045 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:36:30.237012 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086e7cd3_ff89_4558_8199_8fa2a6b7e326.slice/crio-e17712be29528b864fc66f4bb133d71f340b5768545968b8e066f70a95097f6c WatchSource:0}: Error finding container e17712be29528b864fc66f4bb133d71f340b5768545968b8e066f70a95097f6c: Status 404 returned error can't find the container with id e17712be29528b864fc66f4bb133d71f340b5768545968b8e066f70a95097f6c Oct 30 01:36:30.240455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:30.240416 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:36:30.240544 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:30.240505 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:36:30.240586 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:30.240548 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:36:30.392111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:30.392077 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" event={"ID":"086e7cd3-ff89-4558-8199-8fa2a6b7e326","Type":"ContainerStarted","Data":"7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8"} Oct 30 01:36:30.392309 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:30.392139 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" event={"ID":"086e7cd3-ff89-4558-8199-8fa2a6b7e326","Type":"ContainerStarted","Data":"e17712be29528b864fc66f4bb133d71f340b5768545968b8e066f70a95097f6c"} Oct 30 01:36:30.444587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:30.444530 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" podStartSLOduration=1.4445100370000001 podStartE2EDuration="1.444510037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:30.441606923 +0000 UTC m=+1157.762563965" watchObservedRunningTime="2026-10-30 01:36:30.444510037 +0000 UTC m=+1157.765467076" Oct 30 01:36:30.852859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:30.852831 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:31.857933 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:31.857905 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:32.079408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:32.079369 5055 patch_prober.go:29] interesting pod/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:36:32.079565 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:32.079429 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:36:32.418961 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:32.418931 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:32.420964 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:32.420937 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:36:33.863994 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.863968 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:36:33.983108 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983069 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-credential-socket\") pod \"86c97571-2fe7-4f32-bade-7a725d5fce15\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " Oct 30 01:36:33.983279 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983241 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-data\") pod \"86c97571-2fe7-4f32-bade-7a725d5fce15\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " Oct 30 01:36:33.983392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983294 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-kube-api-access-5jrzg\" (UniqueName: \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-kube-api-access-5jrzg\") pod \"86c97571-2fe7-4f32-bade-7a725d5fce15\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " Oct 30 01:36:33.983392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983321 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-podinfo\") pod \"86c97571-2fe7-4f32-bade-7a725d5fce15\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " Oct 30 01:36:33.983521 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983402 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-credential-socket" pod "86c97571-2fe7-4f32-bade-7a725d5fce15" (UID: "86c97571-2fe7-4f32-bade-7a725d5fce15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:33.983521 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983465 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-token\" (UniqueName: \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-token\") pod \"86c97571-2fe7-4f32-bade-7a725d5fce15\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " Oct 30 01:36:33.983630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983529 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-envoy\") pod \"86c97571-2fe7-4f32-bade-7a725d5fce15\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " Oct 30 01:36:33.983630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983552 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-data" pod "86c97571-2fe7-4f32-bade-7a725d5fce15" (UID: "86c97571-2fe7-4f32-bade-7a725d5fce15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:33.983630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983597 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-certs\") pod \"86c97571-2fe7-4f32-bade-7a725d5fce15\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " Oct 30 01:36:33.983785 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983628 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-socket\") pod \"86c97571-2fe7-4f32-bade-7a725d5fce15\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " Oct 30 01:36:33.983785 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.983742 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c97571-2fe7-4f32-bade-7a725d5fce15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c97571-2fe7-4f32-bade-7a725d5fce15-istiod-ca-cert\") pod \"86c97571-2fe7-4f32-bade-7a725d5fce15\" (UID: \"86c97571-2fe7-4f32-bade-7a725d5fce15\") " Oct 30 01:36:33.984816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.984278 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-certs" pod "86c97571-2fe7-4f32-bade-7a725d5fce15" (UID: "86c97571-2fe7-4f32-bade-7a725d5fce15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:33.984816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.984465 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-socket" pod "86c97571-2fe7-4f32-bade-7a725d5fce15" (UID: "86c97571-2fe7-4f32-bade-7a725d5fce15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:33.984816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.984744 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:33.984816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.984763 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:33.984816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.984778 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:33.984816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.984780 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c97571-2fe7-4f32-bade-7a725d5fce15-istiod-ca-cert" pod "86c97571-2fe7-4f32-bade-7a725d5fce15" (UID: "86c97571-2fe7-4f32-bade-7a725d5fce15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:33.984816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.984792 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:33.986118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.986020 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-kube-api-access-5jrzg" pod "86c97571-2fe7-4f32-bade-7a725d5fce15" (UID: "86c97571-2fe7-4f32-bade-7a725d5fce15"). InnerVolumeSpecName "kube-api-access-5jrzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:33.987596 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.987574 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-token" pod "86c97571-2fe7-4f32-bade-7a725d5fce15" (UID: "86c97571-2fe7-4f32-bade-7a725d5fce15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:33.988007 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.987978 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-envoy" pod "86c97571-2fe7-4f32-bade-7a725d5fce15" (UID: "86c97571-2fe7-4f32-bade-7a725d5fce15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:33.990708 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:33.990686 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-podinfo" pod "86c97571-2fe7-4f32-bade-7a725d5fce15" (UID: "86c97571-2fe7-4f32-bade-7a725d5fce15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:36:34.086547 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.086466 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jrzg\" (UniqueName: \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-kube-api-access-5jrzg\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:34.086547 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.086496 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:34.086547 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.086508 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:34.086547 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.086521 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86c97571-2fe7-4f32-bade-7a725d5fce15-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:34.086547 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.086530 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86c97571-2fe7-4f32-bade-7a725d5fce15-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:36:34.434675 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.434583 5055 generic.go:356] "Generic (PLEG): container finished" podID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerID="0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7" exitCode=0 Oct 30 01:36:34.434675 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.434633 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" Oct 30 01:36:34.434675 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.434665 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" event={"ID":"86c97571-2fe7-4f32-bade-7a725d5fce15","Type":"ContainerDied","Data":"0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7"} Oct 30 01:36:34.434931 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.434722 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj" event={"ID":"86c97571-2fe7-4f32-bade-7a725d5fce15","Type":"ContainerDied","Data":"8a42fef3dc6b6bd3b6349fb9893e0927b17045a4292425ca777ca5cb0552fb3b"} Oct 30 01:36:34.434931 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.434753 5055 scope.go:122] "RemoveContainer" containerID="0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7" Oct 30 01:36:34.446213 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.446099 5055 scope.go:122] "RemoveContainer" containerID="8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab" Oct 30 01:36:34.455997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.455977 5055 scope.go:122] "RemoveContainer" containerID="0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7" Oct 30 01:36:34.456253 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:36:34.456234 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7\": container with ID starting with 0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7 not found: ID does not exist" containerID="0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7" Oct 30 01:36:34.456298 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.456263 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7"} err="failed to get container status \"0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7\": rpc error: code = NotFound desc = could not find container \"0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7\": container with ID starting with 0e0e3b356f7977601ab5bf27987fd233b9f06018755d45ebcabd86355888c3f7 not found: ID does not exist" Oct 30 01:36:34.456298 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.456281 5055 scope.go:122] "RemoveContainer" containerID="8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab" Oct 30 01:36:34.456584 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:36:34.456566 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab\": container with ID starting with 8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab not found: ID does not exist" containerID="8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab" Oct 30 01:36:34.456625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.456590 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab"} err="failed to get container status \"8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab\": rpc error: code = NotFound desc = could not find container \"8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab\": container with ID starting with 8ab3022fa286983c6a0f1fa60ad6a0f9e03e8ab654f83b163fc3cf67eaa4e0ab not found: ID does not exist" Oct 30 01:36:34.502802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.502768 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj"] Oct 30 01:36:34.513875 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.513846 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a06822-openshift-default-6954d7db7f-xwqgj"] Oct 30 01:36:34.956143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:34.953207 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" path="/var/lib/kubelet/pods/86c97571-2fe7-4f32-bade-7a725d5fce15/volumes" Oct 30 01:36:46.793486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.790120 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9"] Oct 30 01:36:46.793486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.791116 5055 cpu_manager.go:414] "containerMap: removing container" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" Oct 30 01:36:46.793486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.791135 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" Oct 30 01:36:46.793486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.791152 5055 cpu_manager.go:414] "containerMap: removing container" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" Oct 30 01:36:46.793486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.791158 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" Oct 30 01:36:46.793486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.791293 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" Oct 30 01:36:46.793486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.791309 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c97571-2fe7-4f32-bade-7a725d5fce15" containerName="istio-proxy" Oct 30 01:36:46.796059 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.796040 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:46.822352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.820886 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f09934-serving-cert\"" Oct 30 01:36:46.829099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.829073 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9"] Oct 30 01:36:46.903744 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.903709 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-maas-api-tls\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:46.903870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.903767 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-openshift-service-ca\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:46.903978 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.903958 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbhv\" (UniqueName: \"kubernetes.io/projected/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-kube-api-access-xzbhv\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:46.904111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:46.904000 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-metrics-certs\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:47.005295 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.005267 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-maas-api-tls\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:47.005467 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.005364 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-openshift-service-ca\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:47.005525 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.005497 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbhv\" (UniqueName: \"kubernetes.io/projected/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-kube-api-access-xzbhv\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:47.005603 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.005567 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-metrics-certs\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:47.006109 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.006087 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-openshift-service-ca\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:47.008255 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.008225 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-metrics-certs\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:47.008388 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.008315 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-maas-api-tls\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:47.025296 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.025275 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbhv\" (UniqueName: \"kubernetes.io/projected/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-kube-api-access-xzbhv\") pod \"maas-api-e2e-api-a-f09934-865f8b78fd-77jd9\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:47.134232 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.134130 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:47.454976 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:36:47.454945 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6329c7b5_a9ae_4113_aceb_be3fbb71ab35.slice/crio-993deba4f6167d20a8cfa9931a00242f718b89bfc09841a73f807a5da11cd0d0 WatchSource:0}: Error finding container 993deba4f6167d20a8cfa9931a00242f718b89bfc09841a73f807a5da11cd0d0: Status 404 returned error can't find the container with id 993deba4f6167d20a8cfa9931a00242f718b89bfc09841a73f807a5da11cd0d0 Oct 30 01:36:47.456579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.456549 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9"] Oct 30 01:36:47.582775 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:47.582734 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" event={"ID":"6329c7b5-a9ae-4113-aceb-be3fbb71ab35","Type":"ContainerStarted","Data":"993deba4f6167d20a8cfa9931a00242f718b89bfc09841a73f807a5da11cd0d0"} Oct 30 01:36:48.590879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:48.590842 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" event={"ID":"6329c7b5-a9ae-4113-aceb-be3fbb71ab35","Type":"ContainerStarted","Data":"a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9"} Oct 30 01:36:48.591315 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:48.591100 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:36:48.631021 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:48.630964 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" podStartSLOduration=2.093750329 podStartE2EDuration="2.630945638s" totalImagesPullingTime="537.195309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:48.629463977 +0000 UTC m=+1175.950421032" watchObservedRunningTime="2026-10-30 01:36:48.630945638 +0000 UTC m=+1175.951902680" Oct 30 01:36:49.219558 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:36:49.219521 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:36:54.604886 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:36:54.604856 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:37:03.459492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:03.459453 5055 patch_prober.go:29] interesting pod/e2e-api-a-f09934-openshift-default-748c86d47-27bdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:37:03.460016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:03.459534 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:37:05.778681 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:05.778648 5055 generic.go:356] "Generic (PLEG): container finished" podID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerID="7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8" exitCode=0 Oct 30 01:37:05.779173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:05.778707 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" event={"ID":"086e7cd3-ff89-4558-8199-8fa2a6b7e326","Type":"ContainerDied","Data":"7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8"} Oct 30 01:37:05.779264 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:05.779247 5055 scope.go:122] "RemoveContainer" containerID="7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8" Oct 30 01:37:05.781009 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:05.780975 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:05.781135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:05.781069 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:05.781135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:05.781114 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:06.061643 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.061611 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg"] Oct 30 01:37:06.061899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.061846 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" podUID="b79ffe20-20bc-4e7d-9298-e7cb8d94d930" containerName="maas-api" containerID="cri-o://c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3" gracePeriod=30 Oct 30 01:37:06.666829 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.666800 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:37:06.798030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.797931 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-openshift-service-ca\") pod \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " Oct 30 01:37:06.800542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.800500 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-metrics-certs\") pod \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " Oct 30 01:37:06.800762 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.800747 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-maas-api-tls\") pod \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " Oct 30 01:37:06.800939 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.800924 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-kube-api-access-gpwkb\" (UniqueName: \"kubernetes.io/projected/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-kube-api-access-gpwkb\") pod \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\" (UID: \"b79ffe20-20bc-4e7d-9298-e7cb8d94d930\") " Oct 30 01:37:06.808389 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.808316 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-openshift-service-ca" pod "b79ffe20-20bc-4e7d-9298-e7cb8d94d930" (UID: "b79ffe20-20bc-4e7d-9298-e7cb8d94d930"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:06.810690 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.810647 5055 generic.go:356] "Generic (PLEG): container finished" podID="b79ffe20-20bc-4e7d-9298-e7cb8d94d930" containerID="c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3" exitCode=0 Oct 30 01:37:06.812153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.811089 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" Oct 30 01:37:06.815437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.811825 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" event={"ID":"b79ffe20-20bc-4e7d-9298-e7cb8d94d930","Type":"ContainerDied","Data":"c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3"} Oct 30 01:37:06.815617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.815593 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg" event={"ID":"b79ffe20-20bc-4e7d-9298-e7cb8d94d930","Type":"ContainerDied","Data":"cc3dc24cdb31a756bc73b8cc34e8d8cd5a105bd9e9a62b5d2502dfad843eb4a9"} Oct 30 01:37:06.818721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.815713 5055 scope.go:122] "RemoveContainer" containerID="c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3" Oct 30 01:37:06.820538 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.820504 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-kube-api-access-gpwkb" pod "b79ffe20-20bc-4e7d-9298-e7cb8d94d930" (UID: "b79ffe20-20bc-4e7d-9298-e7cb8d94d930"). InnerVolumeSpecName "kube-api-access-gpwkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:06.850763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.850724 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-maas-api-tls" pod "b79ffe20-20bc-4e7d-9298-e7cb8d94d930" (UID: "b79ffe20-20bc-4e7d-9298-e7cb8d94d930"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:37:06.851831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.851800 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-metrics-certs" pod "b79ffe20-20bc-4e7d-9298-e7cb8d94d930" (UID: "b79ffe20-20bc-4e7d-9298-e7cb8d94d930"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:37:06.852866 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.852836 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" event={"ID":"086e7cd3-ff89-4558-8199-8fa2a6b7e326","Type":"ContainerStarted","Data":"fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6"} Oct 30 01:37:06.853975 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.853953 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:37:06.854124 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.854100 5055 patch_prober.go:29] interesting pod/e2e-api-a-f09934-openshift-default-748c86d47-27bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:37:06.854196 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.854149 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:37:06.884650 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.884250 5055 scope.go:122] "RemoveContainer" containerID="c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3" Oct 30 01:37:06.885003 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:37:06.884903 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3\": container with ID starting with c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3 not found: ID does not exist" containerID="c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3" Oct 30 01:37:06.885003 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.884941 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3"} err="failed to get container status \"c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3\": rpc error: code = NotFound desc = could not find container \"c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3\": container with ID starting with c325e917ed9a7d6a24fd120afdb43e9afd041145db9725eb3ca80f0da68cf7f3 not found: ID does not exist" Oct 30 01:37:06.909364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.908541 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:06.909364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.908575 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:06.909364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.908589 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:06.909364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:06.908603 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpwkb\" (UniqueName: \"kubernetes.io/projected/b79ffe20-20bc-4e7d-9298-e7cb8d94d930-kube-api-access-gpwkb\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:07.195198 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:07.193377 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg"] Oct 30 01:37:07.199772 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:07.199703 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-97c137-6dc57d588-ppllg"] Oct 30 01:37:07.853255 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:07.853218 5055 patch_prober.go:29] interesting pod/e2e-api-a-f09934-openshift-default-748c86d47-27bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:37:07.853671 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:07.853269 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:37:08.908351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:08.902663 5055 patch_prober.go:29] interesting pod/e2e-api-a-f09934-openshift-default-748c86d47-27bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:37:08.993689 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:08.993651 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:37:08.994471 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:08.974991 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79ffe20-20bc-4e7d-9298-e7cb8d94d930" path="/var/lib/kubelet/pods/b79ffe20-20bc-4e7d-9298-e7cb8d94d930/volumes" Oct 30 01:37:09.314456 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.314409 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf"] Oct 30 01:37:09.315302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.315269 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" containerID="cri-o://195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae" gracePeriod=30 Oct 30 01:37:09.758611 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.758455 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg"] Oct 30 01:37:09.765623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.759471 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b79ffe20-20bc-4e7d-9298-e7cb8d94d930" containerName="maas-api" Oct 30 01:37:09.765623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.759495 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79ffe20-20bc-4e7d-9298-e7cb8d94d930" containerName="maas-api" Oct 30 01:37:09.765623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.759709 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79ffe20-20bc-4e7d-9298-e7cb8d94d930" containerName="maas-api" Oct 30 01:37:09.765623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.763777 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:09.793172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.793135 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg"] Oct 30 01:37:09.804352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.802602 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-18560b-openshift-default-dockercfg-dm5gj\"" Oct 30 01:37:09.862356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.852812 5055 patch_prober.go:29] interesting pod/e2e-api-a-f09934-openshift-default-748c86d47-27bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:37:09.862659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.862627 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:37:09.982341 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.979138 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-certs\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:09.982341 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:09.979287 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-podinfo\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.031670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.031585 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-data\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.032006 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.031989 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-socket\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.032117 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.032102 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gmtp\" (UniqueName: \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-kube-api-access-2gmtp\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.032225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.032213 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-credential-socket\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.032411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.032395 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f0697a-4001-46eb-8bf6-83200b8d0062-istiod-ca-cert\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.053269 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.053234 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-envoy\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.062349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.053514 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-token\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.172513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.172479 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-socket\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.173169 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.173148 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gmtp\" (UniqueName: \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-kube-api-access-2gmtp\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.173739 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.173719 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-credential-socket\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.174177 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.174157 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f0697a-4001-46eb-8bf6-83200b8d0062-istiod-ca-cert\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.174311 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.174295 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-envoy\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.174449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.174434 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-token\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.174728 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.174708 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-certs\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.176294 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.176275 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-podinfo\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.176879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.173087 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-socket\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.177014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.175586 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-certs\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.177133 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.174114 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-credential-socket\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.177407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.177389 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-data\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.177858 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.177757 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-data\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.178719 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.178131 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f0697a-4001-46eb-8bf6-83200b8d0062-istiod-ca-cert\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.211963 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.211928 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-podinfo\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.213951 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.213918 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gmtp\" (UniqueName: \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-kube-api-access-2gmtp\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.215628 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.215607 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-envoy\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.215993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.215975 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-token\") pod \"e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.413571 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.413463 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:10.639234 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.637844 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d"] Oct 30 01:37:10.639731 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.639697 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" podUID="0ad750c7-80da-4d24-af92-37b066980ff0" containerName="maas-api" containerID="cri-o://c7e31ed34912dc396623fde0221708db693f9d5b8f3611fc4eb26be8d7dad28a" gracePeriod=30 Oct 30 01:37:10.806783 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:37:10.806747 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f0697a_4001_46eb_8bf6_83200b8d0062.slice/crio-596db85e61879c1398635c279d5a27f04b0cb568350bb69ff300650d97b5871b WatchSource:0}: Error finding container 596db85e61879c1398635c279d5a27f04b0cb568350bb69ff300650d97b5871b: Status 404 returned error can't find the container with id 596db85e61879c1398635c279d5a27f04b0cb568350bb69ff300650d97b5871b Oct 30 01:37:10.807974 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.807924 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg"] Oct 30 01:37:10.815778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.814873 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:10.815778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.814966 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:10.815778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.815012 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:10.852788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.852758 5055 patch_prober.go:29] interesting pod/e2e-api-a-f09934-openshift-default-748c86d47-27bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:37:10.853025 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:10.853001 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:37:11.549439 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.540581 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:37:11.629602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.629567 5055 generic.go:356] "Generic (PLEG): container finished" podID="0ad750c7-80da-4d24-af92-37b066980ff0" containerID="c7e31ed34912dc396623fde0221708db693f9d5b8f3611fc4eb26be8d7dad28a" exitCode=0 Oct 30 01:37:11.629981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.629818 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" event={"ID":"0ad750c7-80da-4d24-af92-37b066980ff0","Type":"ContainerDied","Data":"c7e31ed34912dc396623fde0221708db693f9d5b8f3611fc4eb26be8d7dad28a"} Oct 30 01:37:11.630125 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.630115 5055 scope.go:122] "RemoveContainer" containerID="c7e31ed34912dc396623fde0221708db693f9d5b8f3611fc4eb26be8d7dad28a" Oct 30 01:37:11.630281 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.629885 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d" Oct 30 01:37:11.634764 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.634742 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" event={"ID":"f7f0697a-4001-46eb-8bf6-83200b8d0062","Type":"ContainerStarted","Data":"355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565"} Oct 30 01:37:11.634901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.634889 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" event={"ID":"f7f0697a-4001-46eb-8bf6-83200b8d0062","Type":"ContainerStarted","Data":"596db85e61879c1398635c279d5a27f04b0cb568350bb69ff300650d97b5871b"} Oct 30 01:37:11.656993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.654704 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ad750c7-80da-4d24-af92-37b066980ff0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad750c7-80da-4d24-af92-37b066980ff0-openshift-service-ca\") pod \"0ad750c7-80da-4d24-af92-37b066980ff0\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " Oct 30 01:37:11.656993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.654853 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-maas-api-tls\") pod \"0ad750c7-80da-4d24-af92-37b066980ff0\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " Oct 30 01:37:11.656993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.654890 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ad750c7-80da-4d24-af92-37b066980ff0-kube-api-access-pzqfb\" (UniqueName: \"kubernetes.io/projected/0ad750c7-80da-4d24-af92-37b066980ff0-kube-api-access-pzqfb\") pod \"0ad750c7-80da-4d24-af92-37b066980ff0\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " Oct 30 01:37:11.656993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.655085 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-metrics-certs\") pod \"0ad750c7-80da-4d24-af92-37b066980ff0\" (UID: \"0ad750c7-80da-4d24-af92-37b066980ff0\") " Oct 30 01:37:11.658644 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.658604 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad750c7-80da-4d24-af92-37b066980ff0-openshift-service-ca" pod "0ad750c7-80da-4d24-af92-37b066980ff0" (UID: "0ad750c7-80da-4d24-af92-37b066980ff0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:11.660968 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.660939 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-metrics-certs" pod "0ad750c7-80da-4d24-af92-37b066980ff0" (UID: "0ad750c7-80da-4d24-af92-37b066980ff0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:37:11.691618 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.691506 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad750c7-80da-4d24-af92-37b066980ff0-kube-api-access-pzqfb" pod "0ad750c7-80da-4d24-af92-37b066980ff0" (UID: "0ad750c7-80da-4d24-af92-37b066980ff0"). InnerVolumeSpecName "kube-api-access-pzqfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:11.691618 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.691590 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-maas-api-tls" pod "0ad750c7-80da-4d24-af92-37b066980ff0" (UID: "0ad750c7-80da-4d24-af92-37b066980ff0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:37:11.718649 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.718596 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" podStartSLOduration=2.718575214 podStartE2EDuration="2.718575214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:11.696452614 +0000 UTC m=+1199.017409655" watchObservedRunningTime="2026-10-30 01:37:11.718575214 +0000 UTC m=+1199.039532292" Oct 30 01:37:11.758099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.757521 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:11.758099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.757565 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ad750c7-80da-4d24-af92-37b066980ff0-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:11.758099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.757579 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ad750c7-80da-4d24-af92-37b066980ff0-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:11.758099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.757592 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqfb\" (UniqueName: \"kubernetes.io/projected/0ad750c7-80da-4d24-af92-37b066980ff0-kube-api-access-pzqfb\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:11.854670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.854588 5055 patch_prober.go:29] interesting pod/e2e-api-a-f09934-openshift-default-748c86d47-27bdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body= Oct 30 01:37:11.854919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:11.854888 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" Oct 30 01:37:12.104363 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.103644 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d"] Oct 30 01:37:12.104363 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.104000 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-288c8f-7b876765dd-cq82d"] Oct 30 01:37:12.414275 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.414223 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:12.420493 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.420432 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:12.646486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.646417 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:12.647700 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.647680 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:12.837661 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.837628 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ad750c7-80da-4d24-af92-37b066980ff0" path="/var/lib/kubelet/pods/0ad750c7-80da-4d24-af92-37b066980ff0/volumes" Oct 30 01:37:12.913996 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.913905 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:37:12.918347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.915275 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:37:12.918347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.917550 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:37:12.997229 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.997195 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:37:12.998561 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:12.998539 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:37:13.332198 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:13.332163 5055 scope.go:122] "RemoveContainer" containerID="e7dd8b1b2466638a7718c5ac231a50b4ac1f769b37fa254a2f2719fc3194c856" Oct 30 01:37:14.637106 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.637043 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:37:14.663790 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.663752 5055 generic.go:356] "Generic (PLEG): container finished" podID="5743d668-2053-4276-a420-fd8a77559a01" containerID="195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae" exitCode=0 Oct 30 01:37:14.663790 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.663784 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" event={"ID":"5743d668-2053-4276-a420-fd8a77559a01","Type":"ContainerDied","Data":"195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae"} Oct 30 01:37:14.663790 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.663805 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" Oct 30 01:37:14.664172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.663998 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" event={"ID":"5743d668-2053-4276-a420-fd8a77559a01","Type":"ContainerDied","Data":"d4cb4137e50d7fe8c9005797e83a5a07d44eec552562156c834370ca6b95ce3b"} Oct 30 01:37:14.664172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.664037 5055 scope.go:122] "RemoveContainer" containerID="195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae" Oct 30 01:37:14.681046 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.680594 5055 scope.go:122] "RemoveContainer" containerID="195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae" Oct 30 01:37:14.681046 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:37:14.680896 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae\": container with ID starting with 195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae not found: ID does not exist" containerID="195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae" Oct 30 01:37:14.681046 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.680922 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae"} err="failed to get container status \"195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae\": rpc error: code = NotFound desc = could not find container \"195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae\": container with ID starting with 195ebcca2a3362a4afaf55f2754a5d853446a9069e8ecf9a55b9357601689fae not found: ID does not exist" Oct 30 01:37:14.752305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.751588 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-envoy\") pod \"5743d668-2053-4276-a420-fd8a77559a01\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " Oct 30 01:37:14.752305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.751755 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5743d668-2053-4276-a420-fd8a77559a01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5743d668-2053-4276-a420-fd8a77559a01-istio-podinfo\") pod \"5743d668-2053-4276-a420-fd8a77559a01\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " Oct 30 01:37:14.752305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.751817 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-certs\") pod \"5743d668-2053-4276-a420-fd8a77559a01\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " Oct 30 01:37:14.752305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.751935 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-kube-api-access-v4sm5\" (UniqueName: \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-kube-api-access-v4sm5\") pod \"5743d668-2053-4276-a420-fd8a77559a01\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " Oct 30 01:37:14.752305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.751989 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-data\") pod \"5743d668-2053-4276-a420-fd8a77559a01\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " Oct 30 01:37:14.752305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.752044 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-credential-socket\") pod \"5743d668-2053-4276-a420-fd8a77559a01\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " Oct 30 01:37:14.752305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.752082 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5743d668-2053-4276-a420-fd8a77559a01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5743d668-2053-4276-a420-fd8a77559a01-istiod-ca-cert\") pod \"5743d668-2053-4276-a420-fd8a77559a01\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " Oct 30 01:37:14.752305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.752193 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-istio-token\" (UniqueName: \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-istio-token\") pod \"5743d668-2053-4276-a420-fd8a77559a01\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " Oct 30 01:37:14.752305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.752264 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-socket\") pod \"5743d668-2053-4276-a420-fd8a77559a01\" (UID: \"5743d668-2053-4276-a420-fd8a77559a01\") " Oct 30 01:37:14.754276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.754244 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-socket" pod "5743d668-2053-4276-a420-fd8a77559a01" (UID: "5743d668-2053-4276-a420-fd8a77559a01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:14.756065 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.756033 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-certs" pod "5743d668-2053-4276-a420-fd8a77559a01" (UID: "5743d668-2053-4276-a420-fd8a77559a01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:14.756817 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.756784 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-credential-socket" pod "5743d668-2053-4276-a420-fd8a77559a01" (UID: "5743d668-2053-4276-a420-fd8a77559a01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:14.757175 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.757147 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5743d668-2053-4276-a420-fd8a77559a01-istiod-ca-cert" pod "5743d668-2053-4276-a420-fd8a77559a01" (UID: "5743d668-2053-4276-a420-fd8a77559a01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:14.757475 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.757447 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-data" pod "5743d668-2053-4276-a420-fd8a77559a01" (UID: "5743d668-2053-4276-a420-fd8a77559a01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:14.758614 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.758528 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-kube-api-access-v4sm5" pod "5743d668-2053-4276-a420-fd8a77559a01" (UID: "5743d668-2053-4276-a420-fd8a77559a01"). InnerVolumeSpecName "kube-api-access-v4sm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:14.758614 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.758528 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5743d668-2053-4276-a420-fd8a77559a01-istio-podinfo" pod "5743d668-2053-4276-a420-fd8a77559a01" (UID: "5743d668-2053-4276-a420-fd8a77559a01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:37:14.760090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.760060 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-istio-token" pod "5743d668-2053-4276-a420-fd8a77559a01" (UID: "5743d668-2053-4276-a420-fd8a77559a01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:14.761999 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.761966 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-envoy" pod "5743d668-2053-4276-a420-fd8a77559a01" (UID: "5743d668-2053-4276-a420-fd8a77559a01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:14.855087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.855023 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.855087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.855053 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.855087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.855068 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5743d668-2053-4276-a420-fd8a77559a01-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.855087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.855081 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.855302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.855093 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.855302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.855105 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.855302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.855118 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5743d668-2053-4276-a420-fd8a77559a01-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.855302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.855129 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5743d668-2053-4276-a420-fd8a77559a01-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:14.855302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:14.855143 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4sm5\" (UniqueName: \"kubernetes.io/projected/5743d668-2053-4276-a420-fd8a77559a01-kube-api-access-v4sm5\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:37:15.007303 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:15.007269 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf"] Oct 30 01:37:15.012410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:15.012372 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf"] Oct 30 01:37:15.515274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:15.515241 5055 patch_prober.go:29] interesting pod/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:37:15.515499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:15.515297 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-278fe7d5-openshift-default-85ddb44945-m9hkf" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:37:16.820921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:16.820880 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5743d668-2053-4276-a420-fd8a77559a01" path="/var/lib/kubelet/pods/5743d668-2053-4276-a420-fd8a77559a01/volumes" Oct 30 01:37:19.220423 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:37:19.220380 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:37:30.029952 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.029860 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz"] Oct 30 01:37:30.041131 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.041100 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6a97281-xstnz"] Oct 30 01:37:30.822641 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.822604 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1" path="/var/lib/kubelet/pods/f3e54b25-6a8c-48e6-8eb8-68ebdd76d1d1/volumes" Oct 30 01:37:30.922980 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.922946 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255"] Oct 30 01:37:30.924111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.924082 5055 cpu_manager.go:414] "containerMap: removing container" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" Oct 30 01:37:30.924272 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.924258 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" Oct 30 01:37:30.924429 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.924415 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0ad750c7-80da-4d24-af92-37b066980ff0" containerName="maas-api" Oct 30 01:37:30.924531 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.924519 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ad750c7-80da-4d24-af92-37b066980ff0" containerName="maas-api" Oct 30 01:37:30.924805 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.924791 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ad750c7-80da-4d24-af92-37b066980ff0" containerName="maas-api" Oct 30 01:37:30.924915 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.924905 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="5743d668-2053-4276-a420-fd8a77559a01" containerName="istio-proxy" Oct 30 01:37:30.935410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.935387 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:30.958076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.958054 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d9ede0-openshift-default-dockercfg-dqdtd\"" Oct 30 01:37:30.976539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:30.976514 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255"] Oct 30 01:37:31.182517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.176477 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-podinfo\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.182517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.176540 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-certs\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.182517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.176637 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbwx\" (UniqueName: \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-kube-api-access-gzbwx\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.182517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.176701 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-credential-socket\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.182517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.176742 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-data\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.182517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.176800 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-token\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.182517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.176844 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-socket\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.182517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.176869 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-envoy\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.182517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.176958 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f7826b4-dd81-4d91-8872-4079c418b79e-istiod-ca-cert\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.281124 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.281089 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-credential-socket\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.282511 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.282482 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-data\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.282690 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.281788 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-credential-socket\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.282780 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-token\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.282856 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-socket\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.282886 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-envoy\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.282974 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f7826b4-dd81-4d91-8872-4079c418b79e-istiod-ca-cert\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.283120 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-podinfo\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.283168 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-certs\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.283271 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbwx\" (UniqueName: \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-kube-api-access-gzbwx\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.283453 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-data\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284545 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.284161 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f7826b4-dd81-4d91-8872-4079c418b79e-istiod-ca-cert\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284545 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.284429 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-socket\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.284926 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.284906 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-certs\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.290243 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.290221 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-envoy\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.291837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.291816 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-podinfo\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.308363 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.308317 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbwx\" (UniqueName: \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-kube-api-access-gzbwx\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.309099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.309077 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-token\") pod \"e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.554730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.554701 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:31.897707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.897516 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255"] Oct 30 01:37:31.900116 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.900079 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:31.900226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.900165 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:31.900278 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:31.900240 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:32.844946 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:32.844914 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" event={"ID":"7f7826b4-dd81-4d91-8872-4079c418b79e","Type":"ContainerStarted","Data":"4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07"} Oct 30 01:37:32.845446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:32.844955 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" event={"ID":"7f7826b4-dd81-4d91-8872-4079c418b79e","Type":"ContainerStarted","Data":"b28b86262c68c4ca2049c218342ba352db9ad04d3ae2bcd15088abbecfdfbe15"} Oct 30 01:37:32.915683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:32.915640 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" podStartSLOduration=2.915621763 podStartE2EDuration="2.915621763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:32.900621914 +0000 UTC m=+1220.221578954" watchObservedRunningTime="2026-10-30 01:37:32.915621763 +0000 UTC m=+1220.236578805" Oct 30 01:37:33.555485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:33.555449 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:33.562891 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:33.562864 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:33.852435 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:33.852361 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:33.853274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:33.853252 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:37:43.486451 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.486411 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd"] Oct 30 01:37:43.520913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.518585 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.546966 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.546939 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-f7bdcc-openshift-default-dockercfg-45s6p\"" Oct 30 01:37:43.550364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.550337 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd"] Oct 30 01:37:43.668429 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.668386 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-envoy\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.668719 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.668699 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-token\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.668896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.668879 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-socket\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.669012 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.668997 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a299b6-5136-4faa-99e6-4a9d0242869d-istiod-ca-cert\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.669193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.669178 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-certs\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.669480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.669349 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9hp6\" (UniqueName: \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-kube-api-access-s9hp6\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.669660 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.669643 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-credential-socket\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.669817 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.669802 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-podinfo\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.670033 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.670015 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-data\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.773444 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.773355 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-data\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.773696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.773674 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-envoy\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.774058 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.774038 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-token\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.774253 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.774238 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-socket\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.775420 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.775401 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a299b6-5136-4faa-99e6-4a9d0242869d-istiod-ca-cert\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.775623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.775609 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-certs\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.775760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.775748 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9hp6\" (UniqueName: \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-kube-api-access-s9hp6\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.775865 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.775854 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-credential-socket\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.775989 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.775978 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-podinfo\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.785143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.785119 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-envoy\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.785292 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.775365 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-socket\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.785420 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.774570 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-data\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.792119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.792097 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a299b6-5136-4faa-99e6-4a9d0242869d-istiod-ca-cert\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.796509 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.792578 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-certs\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.796620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.793172 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-credential-socket\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.796711 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.793946 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-podinfo\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.803389 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.801758 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-token\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.827548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.826999 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9hp6\" (UniqueName: \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-kube-api-access-s9hp6\") pod \"e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:43.849904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:43.849619 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:44.313157 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:37:44.313121 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a299b6_5136_4faa_99e6_4a9d0242869d.slice/crio-d29b5d7baa75520422db6285c74185d9f41dee2d46d30f2de728411d1a74ff07 WatchSource:0}: Error finding container d29b5d7baa75520422db6285c74185d9f41dee2d46d30f2de728411d1a74ff07: Status 404 returned error can't find the container with id d29b5d7baa75520422db6285c74185d9f41dee2d46d30f2de728411d1a74ff07 Oct 30 01:37:44.322932 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:44.318601 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:44.322932 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:44.318685 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:44.322932 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:44.318728 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:44.323860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:44.323833 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd"] Oct 30 01:37:44.975283 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:44.975250 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" event={"ID":"81a299b6-5136-4faa-99e6-4a9d0242869d","Type":"ContainerStarted","Data":"303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1"} Oct 30 01:37:44.975822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:44.975309 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" event={"ID":"81a299b6-5136-4faa-99e6-4a9d0242869d","Type":"ContainerStarted","Data":"d29b5d7baa75520422db6285c74185d9f41dee2d46d30f2de728411d1a74ff07"} Oct 30 01:37:45.030204 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:45.030154 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" podStartSLOduration=2.030140665 podStartE2EDuration="2.030140665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:45.029358506 +0000 UTC m=+1232.350315546" watchObservedRunningTime="2026-10-30 01:37:45.030140665 +0000 UTC m=+1232.351097704" Oct 30 01:37:45.854352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:45.853765 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:45.872346 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:45.868510 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:45.990689 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:45.990664 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:45.992070 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:45.992048 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:37:49.248660 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:37:49.248625 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:37:52.110128 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.110093 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9"] Oct 30 01:37:52.114901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.114876 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.130312 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.130291 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-d9ede0-serving-cert\"" Oct 30 01:37:52.146920 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.146891 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9"] Oct 30 01:37:52.237573 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.237530 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr7fw\" (UniqueName: \"kubernetes.io/projected/1736cd25-3aa2-4ea0-8557-8f58096599df-kube-api-access-tr7fw\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.237760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.237682 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.237760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.237709 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1736cd25-3aa2-4ea0-8557-8f58096599df-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.237883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.237765 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-metrics-certs\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.348533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.348488 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr7fw\" (UniqueName: \"kubernetes.io/projected/1736cd25-3aa2-4ea0-8557-8f58096599df-kube-api-access-tr7fw\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.350014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.349981 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.351969 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.351941 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1736cd25-3aa2-4ea0-8557-8f58096599df-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.392401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.353946 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1736cd25-3aa2-4ea0-8557-8f58096599df-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.392401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.354385 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-metrics-certs\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.392401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.361878 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-metrics-certs\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.393203 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.393176 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-maas-api-tls\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.414086 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.413988 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr7fw\" (UniqueName: \"kubernetes.io/projected/1736cd25-3aa2-4ea0-8557-8f58096599df-kube-api-access-tr7fw\") pod \"maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.436391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.431063 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:52.801754 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:52.801720 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9"] Oct 30 01:37:52.802362 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:37:52.802290 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1736cd25_3aa2_4ea0_8557_8f58096599df.slice/crio-5214a144bcd1ff12a2c8dd785fd060961cf1dc3361385086599667fe5d39c9f4 WatchSource:0}: Error finding container 5214a144bcd1ff12a2c8dd785fd060961cf1dc3361385086599667fe5d39c9f4: Status 404 returned error can't find the container with id 5214a144bcd1ff12a2c8dd785fd060961cf1dc3361385086599667fe5d39c9f4 Oct 30 01:37:53.069587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:53.069487 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" event={"ID":"1736cd25-3aa2-4ea0-8557-8f58096599df","Type":"ContainerStarted","Data":"5214a144bcd1ff12a2c8dd785fd060961cf1dc3361385086599667fe5d39c9f4"} Oct 30 01:37:54.081272 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:54.081233 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" event={"ID":"1736cd25-3aa2-4ea0-8557-8f58096599df","Type":"ContainerStarted","Data":"9f01bee13c98ad5537115c4aba03f33168d3a5a0293294b15fca9a5963b91bb5"} Oct 30 01:37:54.081794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:54.081390 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:37:54.185834 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:54.185763 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" podStartSLOduration=1.735674227 podStartE2EDuration="2.185744636s" totalImagesPullingTime="450.070409ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:54.184450028 +0000 UTC m=+1241.505407067" watchObservedRunningTime="2026-10-30 01:37:54.185744636 +0000 UTC m=+1241.506701677" Oct 30 01:37:57.103969 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:57.103885 5055 generic.go:356] "Generic (PLEG): container finished" podID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerID="355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565" exitCode=0 Oct 30 01:37:57.103969 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:57.103953 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" event={"ID":"f7f0697a-4001-46eb-8bf6-83200b8d0062","Type":"ContainerDied","Data":"355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565"} Oct 30 01:37:57.104575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:57.104471 5055 scope.go:122] "RemoveContainer" containerID="355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565" Oct 30 01:37:57.106172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:57.106141 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:57.106252 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:57.106212 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:57.106252 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:57.106246 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:37:57.647409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:57.647377 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:58.120371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:58.117508 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" event={"ID":"f7f0697a-4001-46eb-8bf6-83200b8d0062","Type":"ContainerStarted","Data":"b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689"} Oct 30 01:37:58.416138 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:58.414270 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:37:58.419650 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:58.419625 5055 patch_prober.go:29] interesting pod/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:37:58.419782 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:58.419669 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:37:59.414043 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:59.414009 5055 patch_prober.go:29] interesting pod/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:37:59.414450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:37:59.414055 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:38:00.114355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:00.111483 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:38:00.448559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:00.428676 5055 patch_prober.go:29] interesting pod/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:38:00.478351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:00.465399 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:38:01.414213 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:01.414180 5055 patch_prober.go:29] interesting pod/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:38:01.414481 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:01.414242 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:38:02.414772 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:02.414739 5055 patch_prober.go:29] interesting pod/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:38:02.415268 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:02.414803 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:38:03.414558 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:03.414522 5055 patch_prober.go:29] interesting pod/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body= Oct 30 01:38:03.414752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:03.414598 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" Oct 30 01:38:04.418775 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:04.418744 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:38:04.419276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:04.419258 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:38:04.419917 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:04.419899 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:38:19.223288 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:38:19.223246 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:38:19.899861 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:19.899465 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:38:19.899861 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:19.899523 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:21.980160 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:21.978189 5055 generic.go:356] "Generic (PLEG): container finished" podID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerID="4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07" exitCode=0 Oct 30 01:38:21.980160 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:21.978294 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" event={"ID":"7f7826b4-dd81-4d91-8872-4079c418b79e","Type":"ContainerDied","Data":"4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07"} Oct 30 01:38:21.980160 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:21.978832 5055 scope.go:122] "RemoveContainer" containerID="4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07" Oct 30 01:38:21.989823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:21.982141 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:21.989823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:21.982239 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:21.989823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:21.982287 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:22.996950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:22.996917 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" event={"ID":"7f7826b4-dd81-4d91-8872-4079c418b79e","Type":"ContainerStarted","Data":"f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325"} Oct 30 01:38:23.555506 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:23.555481 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:38:23.555682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:23.555609 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:38:23.555682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:23.555654 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:38:24.555204 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:24.555170 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:38:24.555620 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:24.555225 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:38:25.555792 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:25.555762 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:38:25.556178 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:25.555835 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:38:26.555810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:26.555778 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:38:26.556226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:26.555843 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:38:26.733284 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:26.733251 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz"] Oct 30 01:38:26.733597 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:26.733565 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" containerID="cri-o://3cc3494dfb91177b5f35acd69f063b4151f3c3c397a5138b638f8661c7d0a668" gracePeriod=30 Oct 30 01:38:26.746682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:26.746654 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk"] Oct 30 01:38:26.746924 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:26.746881 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerName="istio-proxy" containerID="cri-o://b03e45fb17840c69433ff7b293e259a9947935202de696e3b50d07021720f845" gracePeriod=30 Oct 30 01:38:27.555870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:27.555837 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 01:38:27.556282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:27.555894 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 01:38:28.559647 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:28.559618 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:38:28.560078 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:28.560033 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:38:28.560668 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:28.560647 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:38:32.079792 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.079766 5055 generic.go:356] "Generic (PLEG): container finished" podID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerID="3cc3494dfb91177b5f35acd69f063b4151f3c3c397a5138b638f8661c7d0a668" exitCode=0 Oct 30 01:38:32.080226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.080209 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" event={"ID":"52115e42-4083-4b2a-9e5d-0ac4b0a139f0","Type":"ContainerDied","Data":"3cc3494dfb91177b5f35acd69f063b4151f3c3c397a5138b638f8661c7d0a668"} Oct 30 01:38:32.080403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.080386 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" event={"ID":"52115e42-4083-4b2a-9e5d-0ac4b0a139f0","Type":"ContainerDied","Data":"1d663bf3e8feb1ccc25fc48fe5798557ffa5a36cbf864310eacd8083f08b4780"} Oct 30 01:38:32.084486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.084458 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d663bf3e8feb1ccc25fc48fe5798557ffa5a36cbf864310eacd8083f08b4780" Oct 30 01:38:32.084798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.084635 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" event={"ID":"6d3ec3a7-d8b6-439b-8f10-59222347d5e2","Type":"ContainerDied","Data":"b03e45fb17840c69433ff7b293e259a9947935202de696e3b50d07021720f845"} Oct 30 01:38:32.084798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.084249 5055 generic.go:356] "Generic (PLEG): container finished" podID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerID="b03e45fb17840c69433ff7b293e259a9947935202de696e3b50d07021720f845" exitCode=0 Oct 30 01:38:32.084798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.084753 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" event={"ID":"6d3ec3a7-d8b6-439b-8f10-59222347d5e2","Type":"ContainerDied","Data":"60bc638dfd46f063456f823718762701edccba0f0926779d09930ce335f4d555"} Oct 30 01:38:32.084798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.084780 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60bc638dfd46f063456f823718762701edccba0f0926779d09930ce335f4d555" Oct 30 01:38:32.085681 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.085663 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:38:32.089982 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.088989 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:38:32.154201 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154132 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-podinfo\") pod \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " Oct 30 01:38:32.154413 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154314 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-envoy\") pod \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " Oct 30 01:38:32.154413 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154374 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-kube-api-access-8hbnf\" (UniqueName: \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-kube-api-access-8hbnf\") pod \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " Oct 30 01:38:32.154566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154472 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-data\") pod \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " Oct 30 01:38:32.154566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154511 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-credential-socket\") pod \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " Oct 30 01:38:32.154566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154560 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-token\" (UniqueName: \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-token\") pod \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " Oct 30 01:38:32.154734 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154595 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istiod-ca-cert\") pod \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " Oct 30 01:38:32.154734 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154647 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-certs\") pod \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " Oct 30 01:38:32.154840 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154740 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-certs\") pod \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " Oct 30 01:38:32.154840 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154784 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-token\" (UniqueName: \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-token\") pod \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " Oct 30 01:38:32.154840 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154812 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-kube-api-access-4lxcf\" (UniqueName: \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-kube-api-access-4lxcf\") pod \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " Oct 30 01:38:32.154989 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154860 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-credential-socket\") pod \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " Oct 30 01:38:32.154989 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.154953 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-data\") pod \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " Oct 30 01:38:32.155092 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.155060 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-socket\") pod \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " Oct 30 01:38:32.155143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.155108 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-certs" pod "6d3ec3a7-d8b6-439b-8f10-59222347d5e2" (UID: "6d3ec3a7-d8b6-439b-8f10-59222347d5e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.155143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.155130 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istiod-ca-cert\") pod \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " Oct 30 01:38:32.155316 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.155298 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-envoy\") pod \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " Oct 30 01:38:32.155801 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.155481 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istiod-ca-cert" pod "52115e42-4083-4b2a-9e5d-0ac4b0a139f0" (UID: "52115e42-4083-4b2a-9e5d-0ac4b0a139f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:32.155801 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.155528 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-podinfo\") pod \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\" (UID: \"52115e42-4083-4b2a-9e5d-0ac4b0a139f0\") " Oct 30 01:38:32.155801 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.155665 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-credential-socket" pod "6d3ec3a7-d8b6-439b-8f10-59222347d5e2" (UID: "6d3ec3a7-d8b6-439b-8f10-59222347d5e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.156229 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.155767 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-credential-socket" pod "52115e42-4083-4b2a-9e5d-0ac4b0a139f0" (UID: "52115e42-4083-4b2a-9e5d-0ac4b0a139f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.156229 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.156042 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-socket\") pod \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\" (UID: \"6d3ec3a7-d8b6-439b-8f10-59222347d5e2\") " Oct 30 01:38:32.156825 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.156529 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istiod-ca-cert" pod "6d3ec3a7-d8b6-439b-8f10-59222347d5e2" (UID: "6d3ec3a7-d8b6-439b-8f10-59222347d5e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:32.156825 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.156797 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-socket" pod "6d3ec3a7-d8b6-439b-8f10-59222347d5e2" (UID: "6d3ec3a7-d8b6-439b-8f10-59222347d5e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.157513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.157207 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-data" pod "52115e42-4083-4b2a-9e5d-0ac4b0a139f0" (UID: "52115e42-4083-4b2a-9e5d-0ac4b0a139f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.157513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.157314 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-data" pod "6d3ec3a7-d8b6-439b-8f10-59222347d5e2" (UID: "6d3ec3a7-d8b6-439b-8f10-59222347d5e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.157693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.157607 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.157693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.157628 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.157693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.157643 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.157693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.157656 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.157693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.157669 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.157693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.157683 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.158810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.157696 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.158810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.157709 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.158810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.158059 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-certs" pod "52115e42-4083-4b2a-9e5d-0ac4b0a139f0" (UID: "52115e42-4083-4b2a-9e5d-0ac4b0a139f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.158810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.158636 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-socket" pod "52115e42-4083-4b2a-9e5d-0ac4b0a139f0" (UID: "52115e42-4083-4b2a-9e5d-0ac4b0a139f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.162461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.162439 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-podinfo" pod "52115e42-4083-4b2a-9e5d-0ac4b0a139f0" (UID: "52115e42-4083-4b2a-9e5d-0ac4b0a139f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:32.163573 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.163495 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-envoy" pod "52115e42-4083-4b2a-9e5d-0ac4b0a139f0" (UID: "52115e42-4083-4b2a-9e5d-0ac4b0a139f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.189044 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.189017 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-token" pod "6d3ec3a7-d8b6-439b-8f10-59222347d5e2" (UID: "6d3ec3a7-d8b6-439b-8f10-59222347d5e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:32.189212 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.189093 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-kube-api-access-8hbnf" pod "52115e42-4083-4b2a-9e5d-0ac4b0a139f0" (UID: "52115e42-4083-4b2a-9e5d-0ac4b0a139f0"). InnerVolumeSpecName "kube-api-access-8hbnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:32.189290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.189207 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-envoy" pod "6d3ec3a7-d8b6-439b-8f10-59222347d5e2" (UID: "6d3ec3a7-d8b6-439b-8f10-59222347d5e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.189407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.189385 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-podinfo" pod "6d3ec3a7-d8b6-439b-8f10-59222347d5e2" (UID: "6d3ec3a7-d8b6-439b-8f10-59222347d5e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:32.190144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.190115 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-token" pod "52115e42-4083-4b2a-9e5d-0ac4b0a139f0" (UID: "52115e42-4083-4b2a-9e5d-0ac4b0a139f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:32.191259 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.191232 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-kube-api-access-4lxcf" pod "6d3ec3a7-d8b6-439b-8f10-59222347d5e2" (UID: "6d3ec3a7-d8b6-439b-8f10-59222347d5e2"). InnerVolumeSpecName "kube-api-access-4lxcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:32.259153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.259121 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.259153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.259152 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.259386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.259163 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.259386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.259176 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.259386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.259188 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hbnf\" (UniqueName: \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-kube-api-access-8hbnf\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.259386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.259200 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.259386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.259211 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.259386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.259222 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.259386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.259232 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lxcf\" (UniqueName: \"kubernetes.io/projected/6d3ec3a7-d8b6-439b-8f10-59222347d5e2-kube-api-access-4lxcf\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.259386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.259244 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52115e42-4083-4b2a-9e5d-0ac4b0a139f0-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.723905 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.723868 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb"] Oct 30 01:38:32.724567 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.724543 5055 cpu_manager.go:414] "containerMap: removing container" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" Oct 30 01:38:32.724567 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.724565 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" Oct 30 01:38:32.724724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.724603 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerName="istio-proxy" Oct 30 01:38:32.724724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.724608 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerName="istio-proxy" Oct 30 01:38:32.724724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.724703 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" containerName="istio-proxy" Oct 30 01:38:32.724724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.724718 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" containerName="istio-proxy" Oct 30 01:38:32.729943 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.729926 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.746809 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.746785 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-6b3dbd-serving-cert\"" Oct 30 01:38:32.750468 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.749978 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb"] Oct 30 01:38:32.867107 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.867077 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-metrics-certs\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.867208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.867128 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.867366 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.867350 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvwhh\" (UniqueName: \"kubernetes.io/projected/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-kube-api-access-bvwhh\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.867555 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.867538 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.975299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.969101 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvwhh\" (UniqueName: \"kubernetes.io/projected/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-kube-api-access-bvwhh\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.975299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.969286 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.975299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.969342 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-metrics-certs\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.975299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.969373 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.975299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.970169 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.987119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.987058 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.993161 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.993124 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-metrics-certs\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:32.996412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:32.996380 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvwhh\" (UniqueName: \"kubernetes.io/projected/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-kube-api-access-bvwhh\") pod \"maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:33.043589 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:33.043545 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:33.111379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:33.105213 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk" Oct 30 01:38:33.111379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:33.107012 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz" Oct 30 01:38:33.171530 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:33.171498 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk"] Oct 30 01:38:33.178153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:33.177183 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x2gkk"] Oct 30 01:38:33.207599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:33.207450 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz"] Oct 30 01:38:33.211122 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:33.211095 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-w4cdz"] Oct 30 01:38:33.417926 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:33.417896 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb"] Oct 30 01:38:33.419824 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:38:33.419799 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9380eb7_166e_4602_a7c5_83b8b9e0d84f.slice/crio-c78065f646d1e0c18e77136613b81191cf11fa666ccd3757637a4768afe6b5c7 WatchSource:0}: Error finding container c78065f646d1e0c18e77136613b81191cf11fa666ccd3757637a4768afe6b5c7: Status 404 returned error can't find the container with id c78065f646d1e0c18e77136613b81191cf11fa666ccd3757637a4768afe6b5c7 Oct 30 01:38:34.271409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:34.264134 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" event={"ID":"d9380eb7-166e-4602-a7c5-83b8b9e0d84f","Type":"ContainerStarted","Data":"df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e"} Oct 30 01:38:34.271409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:34.264195 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" event={"ID":"d9380eb7-166e-4602-a7c5-83b8b9e0d84f","Type":"ContainerStarted","Data":"c78065f646d1e0c18e77136613b81191cf11fa666ccd3757637a4768afe6b5c7"} Oct 30 01:38:34.272812 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:34.272551 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:34.325207 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:34.325139 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" podStartSLOduration=1.977386122 podStartE2EDuration="2.325119963s" totalImagesPullingTime="347.733841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:34.321150138 +0000 UTC m=+1281.642107178" watchObservedRunningTime="2026-10-30 01:38:34.325119963 +0000 UTC m=+1281.646077004" Oct 30 01:38:34.548404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:34.548302 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9"] Oct 30 01:38:34.548777 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:34.548661 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" podUID="6329c7b5-a9ae-4113-aceb-be3fbb71ab35" containerName="maas-api" containerID="cri-o://a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9" gracePeriod=30 Oct 30 01:38:34.601691 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:34.601637 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" podUID="6329c7b5-a9ae-4113-aceb-be3fbb71ab35" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.104:8443/health\": dial tcp 10.132.0.104:8443: connect: connection refused" Oct 30 01:38:34.822396 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:34.822246 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52115e42-4083-4b2a-9e5d-0ac4b0a139f0" path="/var/lib/kubelet/pods/52115e42-4083-4b2a-9e5d-0ac4b0a139f0/volumes" Oct 30 01:38:34.823180 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:34.823145 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d3ec3a7-d8b6-439b-8f10-59222347d5e2" path="/var/lib/kubelet/pods/6d3ec3a7-d8b6-439b-8f10-59222347d5e2/volumes" Oct 30 01:38:35.051386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.048232 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:38:35.153900 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.153800 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-maas-api-tls\") pod \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " Oct 30 01:38:35.154284 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.154260 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-metrics-certs\") pod \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " Oct 30 01:38:35.154530 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.154515 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-openshift-service-ca\") pod \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " Oct 30 01:38:35.154721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.154709 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-kube-api-access-xzbhv\" (UniqueName: \"kubernetes.io/projected/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-kube-api-access-xzbhv\") pod \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\" (UID: \"6329c7b5-a9ae-4113-aceb-be3fbb71ab35\") " Oct 30 01:38:35.171349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.166050 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-openshift-service-ca" pod "6329c7b5-a9ae-4113-aceb-be3fbb71ab35" (UID: "6329c7b5-a9ae-4113-aceb-be3fbb71ab35"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:35.217636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.217586 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-kube-api-access-xzbhv" pod "6329c7b5-a9ae-4113-aceb-be3fbb71ab35" (UID: "6329c7b5-a9ae-4113-aceb-be3fbb71ab35"). InnerVolumeSpecName "kube-api-access-xzbhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:35.228658 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.226443 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-metrics-certs" pod "6329c7b5-a9ae-4113-aceb-be3fbb71ab35" (UID: "6329c7b5-a9ae-4113-aceb-be3fbb71ab35"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:35.234486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.234451 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-maas-api-tls" pod "6329c7b5-a9ae-4113-aceb-be3fbb71ab35" (UID: "6329c7b5-a9ae-4113-aceb-be3fbb71ab35"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:35.264931 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.260023 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:35.264931 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.260059 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzbhv\" (UniqueName: \"kubernetes.io/projected/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-kube-api-access-xzbhv\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:35.264931 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.260076 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:35.264931 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.260091 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6329c7b5-a9ae-4113-aceb-be3fbb71ab35-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:35.329349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.328863 5055 generic.go:356] "Generic (PLEG): container finished" podID="6329c7b5-a9ae-4113-aceb-be3fbb71ab35" containerID="a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9" exitCode=0 Oct 30 01:38:35.329848 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.329446 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" Oct 30 01:38:35.330539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.329994 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" event={"ID":"6329c7b5-a9ae-4113-aceb-be3fbb71ab35","Type":"ContainerDied","Data":"a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9"} Oct 30 01:38:35.330539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.330048 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9" event={"ID":"6329c7b5-a9ae-4113-aceb-be3fbb71ab35","Type":"ContainerDied","Data":"993deba4f6167d20a8cfa9931a00242f718b89bfc09841a73f807a5da11cd0d0"} Oct 30 01:38:35.330539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.330076 5055 scope.go:122] "RemoveContainer" containerID="a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9" Oct 30 01:38:35.349356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.348542 5055 scope.go:122] "RemoveContainer" containerID="a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9" Oct 30 01:38:35.349356 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:38:35.348882 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9\": container with ID starting with a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9 not found: ID does not exist" containerID="a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9" Oct 30 01:38:35.349356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.348916 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9"} err="failed to get container status \"a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9\": rpc error: code = NotFound desc = could not find container \"a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9\": container with ID starting with a80fb17862d21b348f88d0fc87f24335cf2bfa8908e7e61242db9585844fb7b9 not found: ID does not exist" Oct 30 01:38:35.396651 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.396617 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9"] Oct 30 01:38:35.407205 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:35.407144 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f09934-865f8b78fd-77jd9"] Oct 30 01:38:36.824836 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:36.824795 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6329c7b5-a9ae-4113-aceb-be3fbb71ab35" path="/var/lib/kubelet/pods/6329c7b5-a9ae-4113-aceb-be3fbb71ab35/volumes" Oct 30 01:38:40.897881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:40.897853 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9"] Oct 30 01:38:40.898304 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:40.898047 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" podUID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" containerName="istio-proxy" containerID="cri-o://335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9" gracePeriod=30 Oct 30 01:38:40.911157 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:40.911128 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g"] Oct 30 01:38:40.913017 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:40.911403 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" containerName="istio-proxy" containerID="cri-o://f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3" gracePeriod=30 Oct 30 01:38:40.916102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:40.916073 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4"] Oct 30 01:38:40.916290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:40.916266 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" containerID="cri-o://94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56" gracePeriod=30 Oct 30 01:38:40.924342 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:40.924299 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp"] Oct 30 01:38:40.925551 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:40.925516 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" containerID="cri-o://2e8ebf104e40759747dfff4bdef8cad508adf388cf0b804683a3bc86df4c46a9" gracePeriod=30 Oct 30 01:38:41.343814 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:41.343783 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:38:43.000604 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:42.984676 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq"] Oct 30 01:38:43.000604 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:42.996566 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" containerID="cri-o://fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6" gracePeriod=30 Oct 30 01:38:46.273189 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.273151 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:38:46.326813 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.326786 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:38:46.337137 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.337110 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:38:46.373012 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.372983 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-token\" (UniqueName: \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-token\") pod \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " Oct 30 01:38:46.373199 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373027 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-credential-socket\") pod \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " Oct 30 01:38:46.373199 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373051 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-certs\") pod \"e9186ef0-4662-4afc-86a5-27d290426e41\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " Oct 30 01:38:46.373199 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373070 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-istio-token\" (UniqueName: \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-istio-token\") pod \"e9186ef0-4662-4afc-86a5-27d290426e41\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " Oct 30 01:38:46.373199 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373117 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-socket\") pod \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " Oct 30 01:38:46.373199 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373159 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-certs\") pod \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " Oct 30 01:38:46.373517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373211 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-credential-socket\") pod \"e9186ef0-4662-4afc-86a5-27d290426e41\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " Oct 30 01:38:46.373517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373299 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-kube-api-access-jgqpv\" (UniqueName: \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-kube-api-access-jgqpv\") pod \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " Oct 30 01:38:46.373517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373340 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-envoy\") pod \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " Oct 30 01:38:46.373517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373376 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-envoy\") pod \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " Oct 30 01:38:46.373517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373405 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-data\") pod \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " Oct 30 01:38:46.373517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373475 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-kube-api-access-6kvjt\" (UniqueName: \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-kube-api-access-6kvjt\") pod \"e9186ef0-4662-4afc-86a5-27d290426e41\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " Oct 30 01:38:46.373517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373511 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-certs\") pod \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " Oct 30 01:38:46.373882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373555 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istiod-ca-cert\") pod \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " Oct 30 01:38:46.373882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373603 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9186ef0-4662-4afc-86a5-27d290426e41-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9186ef0-4662-4afc-86a5-27d290426e41-istiod-ca-cert\") pod \"e9186ef0-4662-4afc-86a5-27d290426e41\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " Oct 30 01:38:46.373882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373631 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-envoy\") pod \"e9186ef0-4662-4afc-86a5-27d290426e41\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " Oct 30 01:38:46.373882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373670 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-data\") pod \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " Oct 30 01:38:46.373882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373698 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-podinfo\") pod \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " Oct 30 01:38:46.373882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373749 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istiod-ca-cert\") pod \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " Oct 30 01:38:46.373882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373794 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9186ef0-4662-4afc-86a5-27d290426e41-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9186ef0-4662-4afc-86a5-27d290426e41-istio-podinfo\") pod \"e9186ef0-4662-4afc-86a5-27d290426e41\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " Oct 30 01:38:46.373882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373834 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-credential-socket\") pod \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373964 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-token\" (UniqueName: \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-token\") pod \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.373992 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-socket\") pod \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\" (UID: \"df22db55-52cd-47ca-8e8c-6ca9db62c9f1\") " Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.374063 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-kube-api-access-w7brn\" (UniqueName: \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-kube-api-access-w7brn\") pod \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.374132 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-data\") pod \"e9186ef0-4662-4afc-86a5-27d290426e41\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.374192 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-socket\") pod \"e9186ef0-4662-4afc-86a5-27d290426e41\" (UID: \"e9186ef0-4662-4afc-86a5-27d290426e41\") " Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.374220 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-podinfo\") pod \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\" (UID: \"8d4eff34-fdc7-40a5-b75d-9b8028822b94\") " Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.374864 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-credential-socket" pod "df22db55-52cd-47ca-8e8c-6ca9db62c9f1" (UID: "df22db55-52cd-47ca-8e8c-6ca9db62c9f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.375065 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.375403 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9186ef0-4662-4afc-86a5-27d290426e41-istiod-ca-cert" pod "e9186ef0-4662-4afc-86a5-27d290426e41" (UID: "e9186ef0-4662-4afc-86a5-27d290426e41"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.375644 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-socket" pod "8d4eff34-fdc7-40a5-b75d-9b8028822b94" (UID: "8d4eff34-fdc7-40a5-b75d-9b8028822b94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.375854 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-certs" pod "8d4eff34-fdc7-40a5-b75d-9b8028822b94" (UID: "8d4eff34-fdc7-40a5-b75d-9b8028822b94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.376100 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-data" pod "8d4eff34-fdc7-40a5-b75d-9b8028822b94" (UID: "8d4eff34-fdc7-40a5-b75d-9b8028822b94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.376103 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-certs" pod "e9186ef0-4662-4afc-86a5-27d290426e41" (UID: "e9186ef0-4662-4afc-86a5-27d290426e41"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.376295 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-certs" pod "df22db55-52cd-47ca-8e8c-6ca9db62c9f1" (UID: "df22db55-52cd-47ca-8e8c-6ca9db62c9f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.376591 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istiod-ca-cert" pod "df22db55-52cd-47ca-8e8c-6ca9db62c9f1" (UID: "df22db55-52cd-47ca-8e8c-6ca9db62c9f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.376626 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-credential-socket" pod "8d4eff34-fdc7-40a5-b75d-9b8028822b94" (UID: "8d4eff34-fdc7-40a5-b75d-9b8028822b94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.376884 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istiod-ca-cert" pod "8d4eff34-fdc7-40a5-b75d-9b8028822b94" (UID: "8d4eff34-fdc7-40a5-b75d-9b8028822b94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.377396 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-credential-socket" pod "e9186ef0-4662-4afc-86a5-27d290426e41" (UID: "e9186ef0-4662-4afc-86a5-27d290426e41"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.377446 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-podinfo" pod "8d4eff34-fdc7-40a5-b75d-9b8028822b94" (UID: "8d4eff34-fdc7-40a5-b75d-9b8028822b94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.377504 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-data" pod "df22db55-52cd-47ca-8e8c-6ca9db62c9f1" (UID: "df22db55-52cd-47ca-8e8c-6ca9db62c9f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.377563 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-token" pod "df22db55-52cd-47ca-8e8c-6ca9db62c9f1" (UID: "df22db55-52cd-47ca-8e8c-6ca9db62c9f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.377636 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-istio-token" pod "e9186ef0-4662-4afc-86a5-27d290426e41" (UID: "e9186ef0-4662-4afc-86a5-27d290426e41"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.377883 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-data" pod "e9186ef0-4662-4afc-86a5-27d290426e41" (UID: "e9186ef0-4662-4afc-86a5-27d290426e41"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.378025 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-socket" pod "df22db55-52cd-47ca-8e8c-6ca9db62c9f1" (UID: "df22db55-52cd-47ca-8e8c-6ca9db62c9f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.378985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.378095 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-socket" pod "e9186ef0-4662-4afc-86a5-27d290426e41" (UID: "e9186ef0-4662-4afc-86a5-27d290426e41"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.405766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.387925 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-kube-api-access-6kvjt" pod "e9186ef0-4662-4afc-86a5-27d290426e41" (UID: "e9186ef0-4662-4afc-86a5-27d290426e41"). InnerVolumeSpecName "kube-api-access-6kvjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:46.405766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.387998 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-envoy" pod "e9186ef0-4662-4afc-86a5-27d290426e41" (UID: "e9186ef0-4662-4afc-86a5-27d290426e41"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.405766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.388096 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-podinfo" pod "df22db55-52cd-47ca-8e8c-6ca9db62c9f1" (UID: "df22db55-52cd-47ca-8e8c-6ca9db62c9f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:46.405766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.388738 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9186ef0-4662-4afc-86a5-27d290426e41-istio-podinfo" pod "e9186ef0-4662-4afc-86a5-27d290426e41" (UID: "e9186ef0-4662-4afc-86a5-27d290426e41"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:46.405766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.389271 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-kube-api-access-jgqpv" pod "df22db55-52cd-47ca-8e8c-6ca9db62c9f1" (UID: "df22db55-52cd-47ca-8e8c-6ca9db62c9f1"). InnerVolumeSpecName "kube-api-access-jgqpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:46.405766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.389999 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-kube-api-access-w7brn" pod "8d4eff34-fdc7-40a5-b75d-9b8028822b94" (UID: "8d4eff34-fdc7-40a5-b75d-9b8028822b94"). InnerVolumeSpecName "kube-api-access-w7brn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:46.405766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.391493 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-envoy" pod "8d4eff34-fdc7-40a5-b75d-9b8028822b94" (UID: "8d4eff34-fdc7-40a5-b75d-9b8028822b94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.405766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.391574 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-token" pod "8d4eff34-fdc7-40a5-b75d-9b8028822b94" (UID: "8d4eff34-fdc7-40a5-b75d-9b8028822b94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:46.405766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.391626 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-envoy" pod "df22db55-52cd-47ca-8e8c-6ca9db62c9f1" (UID: "df22db55-52cd-47ca-8e8c-6ca9db62c9f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.477237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477173 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.477237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477206 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.477237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477221 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.477237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477234 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477245 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477257 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477269 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477281 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477294 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgqpv\" (UniqueName: \"kubernetes.io/projected/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-kube-api-access-jgqpv\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477306 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477318 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477349 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477360 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvjt\" (UniqueName: \"kubernetes.io/projected/e9186ef0-4662-4afc-86a5-27d290426e41-kube-api-access-6kvjt\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477373 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477385 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477397 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9186ef0-4662-4afc-86a5-27d290426e41-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477408 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477419 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477430 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477441 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477453 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9186ef0-4662-4afc-86a5-27d290426e41-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477464 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d4eff34-fdc7-40a5-b75d-9b8028822b94-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477477 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477489 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df22db55-52cd-47ca-8e8c-6ca9db62c9f1-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477501 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7brn\" (UniqueName: \"kubernetes.io/projected/8d4eff34-fdc7-40a5-b75d-9b8028822b94-kube-api-access-w7brn\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.478164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.477515 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9186ef0-4662-4afc-86a5-27d290426e41-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.507659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.505927 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": read tcp 10.132.0.2:47372->10.132.0.107:15021: read: connection reset by peer" start-of-body= Oct 30 01:38:46.507659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.505987 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": read tcp 10.132.0.2:47372->10.132.0.107:15021: read: connection reset by peer" Oct 30 01:38:46.535155 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.535127 5055 generic.go:356] "Generic (PLEG): container finished" podID="09c56a37-9f9e-4635-96db-82383212e92c" containerID="2e8ebf104e40759747dfff4bdef8cad508adf388cf0b804683a3bc86df4c46a9" exitCode=0 Oct 30 01:38:46.535285 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.535264 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" event={"ID":"09c56a37-9f9e-4635-96db-82383212e92c","Type":"ContainerDied","Data":"2e8ebf104e40759747dfff4bdef8cad508adf388cf0b804683a3bc86df4c46a9"} Oct 30 01:38:46.536907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.536884 5055 generic.go:356] "Generic (PLEG): container finished" podID="e9186ef0-4662-4afc-86a5-27d290426e41" containerID="f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3" exitCode=0 Oct 30 01:38:46.537051 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.537037 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" Oct 30 01:38:46.537733 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.537710 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" event={"ID":"e9186ef0-4662-4afc-86a5-27d290426e41","Type":"ContainerDied","Data":"f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3"} Oct 30 01:38:46.537866 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.537794 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g" event={"ID":"e9186ef0-4662-4afc-86a5-27d290426e41","Type":"ContainerDied","Data":"07c6bad224a3c760287d11905e731d667b8d8bd3ff23efcba72a7e7af35d8601"} Oct 30 01:38:46.537866 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.537831 5055 scope.go:122] "RemoveContainer" containerID="f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3" Oct 30 01:38:46.539550 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.539532 5055 generic.go:356] "Generic (PLEG): container finished" podID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerID="94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56" exitCode=0 Oct 30 01:38:46.539645 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.539607 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" event={"ID":"df22db55-52cd-47ca-8e8c-6ca9db62c9f1","Type":"ContainerDied","Data":"94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56"} Oct 30 01:38:46.539645 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.539637 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" event={"ID":"df22db55-52cd-47ca-8e8c-6ca9db62c9f1","Type":"ContainerDied","Data":"170fc886da7449f7d3fea0ca5f26bb58fab3dec771b14a1c0cb96aa42ca9ec50"} Oct 30 01:38:46.539750 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.539690 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4" Oct 30 01:38:46.544592 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.544568 5055 generic.go:356] "Generic (PLEG): container finished" podID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" containerID="335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9" exitCode=0 Oct 30 01:38:46.544687 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.544617 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" Oct 30 01:38:46.544767 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.544749 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" event={"ID":"8d4eff34-fdc7-40a5-b75d-9b8028822b94","Type":"ContainerDied","Data":"335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9"} Oct 30 01:38:46.544825 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.544795 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9" event={"ID":"8d4eff34-fdc7-40a5-b75d-9b8028822b94","Type":"ContainerDied","Data":"6ca88a5dd1eb23f051041dcde290b55530a353daf5d513543737a8042e3658ed"} Oct 30 01:38:46.620657 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.620629 5055 scope.go:122] "RemoveContainer" containerID="f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3" Oct 30 01:38:46.621141 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:38:46.621117 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3\": container with ID starting with f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3 not found: ID does not exist" containerID="f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3" Oct 30 01:38:46.621229 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.621148 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3"} err="failed to get container status \"f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3\": rpc error: code = NotFound desc = could not find container \"f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3\": container with ID starting with f36b790f8df87f719560bdda2a57c04be6289df9715b5ec3a127fb50d4fe59d3 not found: ID does not exist" Oct 30 01:38:46.621229 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.621164 5055 scope.go:122] "RemoveContainer" containerID="94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56" Oct 30 01:38:46.634355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.634299 5055 scope.go:122] "RemoveContainer" containerID="94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56" Oct 30 01:38:46.634711 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:38:46.634671 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56\": container with ID starting with 94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56 not found: ID does not exist" containerID="94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56" Oct 30 01:38:46.634783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.634724 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56"} err="failed to get container status \"94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56\": rpc error: code = NotFound desc = could not find container \"94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56\": container with ID starting with 94f64b9532d913c30105fb76b51689d42e417a940e01f4ffb9282308c1f2dd56 not found: ID does not exist" Oct 30 01:38:46.634783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.634745 5055 scope.go:122] "RemoveContainer" containerID="335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9" Oct 30 01:38:46.644020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.643938 5055 scope.go:122] "RemoveContainer" containerID="335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9" Oct 30 01:38:46.644271 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:38:46.644199 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9\": container with ID starting with 335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9 not found: ID does not exist" containerID="335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9" Oct 30 01:38:46.644271 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.644228 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9"} err="failed to get container status \"335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9\": rpc error: code = NotFound desc = could not find container \"335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9\": container with ID starting with 335698546e1d53766533875cee135943a5ae2930d2825b76ae1a1481d8dbded9 not found: ID does not exist" Oct 30 01:38:46.650249 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.650222 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4"] Oct 30 01:38:46.653562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.653532 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-nm2m4"] Oct 30 01:38:46.672047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.672024 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:38:46.688352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.688319 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g"] Oct 30 01:38:46.692068 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.692036 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-wqk2g"] Oct 30 01:38:46.726922 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.726894 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9"] Oct 30 01:38:46.732797 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.732749 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c5lv9"] Oct 30 01:38:46.785259 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.785231 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-kube-api-access-sb9c7\" (UniqueName: \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-kube-api-access-sb9c7\") pod \"09c56a37-9f9e-4635-96db-82383212e92c\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " Oct 30 01:38:46.785901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.785881 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-envoy\") pod \"09c56a37-9f9e-4635-96db-82383212e92c\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " Oct 30 01:38:46.786107 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.786092 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-credential-socket\") pod \"09c56a37-9f9e-4635-96db-82383212e92c\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " Oct 30 01:38:46.786237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.786224 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-socket\") pod \"09c56a37-9f9e-4635-96db-82383212e92c\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " Oct 30 01:38:46.786414 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.786400 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-certs\") pod \"09c56a37-9f9e-4635-96db-82383212e92c\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " Oct 30 01:38:46.786820 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.786806 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-istio-token\" (UniqueName: \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-istio-token\") pod \"09c56a37-9f9e-4635-96db-82383212e92c\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " Oct 30 01:38:46.786990 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.786977 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09c56a37-9f9e-4635-96db-82383212e92c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c56a37-9f9e-4635-96db-82383212e92c-istio-podinfo\") pod \"09c56a37-9f9e-4635-96db-82383212e92c\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " Oct 30 01:38:46.787375 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.787358 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c56a37-9f9e-4635-96db-82383212e92c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c56a37-9f9e-4635-96db-82383212e92c-istiod-ca-cert\") pod \"09c56a37-9f9e-4635-96db-82383212e92c\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " Oct 30 01:38:46.787601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.787584 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-data\") pod \"09c56a37-9f9e-4635-96db-82383212e92c\" (UID: \"09c56a37-9f9e-4635-96db-82383212e92c\") " Oct 30 01:38:46.788073 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.787091 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-credential-socket" pod "09c56a37-9f9e-4635-96db-82383212e92c" (UID: "09c56a37-9f9e-4635-96db-82383212e92c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.788165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.787489 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-certs" pod "09c56a37-9f9e-4635-96db-82383212e92c" (UID: "09c56a37-9f9e-4635-96db-82383212e92c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.788165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.787603 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-socket" pod "09c56a37-9f9e-4635-96db-82383212e92c" (UID: "09c56a37-9f9e-4635-96db-82383212e92c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.788519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.788480 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c56a37-9f9e-4635-96db-82383212e92c-istiod-ca-cert" pod "09c56a37-9f9e-4635-96db-82383212e92c" (UID: "09c56a37-9f9e-4635-96db-82383212e92c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:46.788519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.788485 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-kube-api-access-sb9c7" pod "09c56a37-9f9e-4635-96db-82383212e92c" (UID: "09c56a37-9f9e-4635-96db-82383212e92c"). InnerVolumeSpecName "kube-api-access-sb9c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:46.788691 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.788669 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-data" pod "09c56a37-9f9e-4635-96db-82383212e92c" (UID: "09c56a37-9f9e-4635-96db-82383212e92c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.788791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.788730 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.788791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.788749 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.788791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.788762 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.788791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.788772 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c56a37-9f9e-4635-96db-82383212e92c-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.788791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.788782 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb9c7\" (UniqueName: \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-kube-api-access-sb9c7\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.793528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.793479 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-envoy" pod "09c56a37-9f9e-4635-96db-82383212e92c" (UID: "09c56a37-9f9e-4635-96db-82383212e92c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:46.793528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.793489 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-istio-token" pod "09c56a37-9f9e-4635-96db-82383212e92c" (UID: "09c56a37-9f9e-4635-96db-82383212e92c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:46.794422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.794391 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09c56a37-9f9e-4635-96db-82383212e92c-istio-podinfo" pod "09c56a37-9f9e-4635-96db-82383212e92c" (UID: "09c56a37-9f9e-4635-96db-82383212e92c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:46.820986 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.820963 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" path="/var/lib/kubelet/pods/8d4eff34-fdc7-40a5-b75d-9b8028822b94/volumes" Oct 30 01:38:46.821449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.821436 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" path="/var/lib/kubelet/pods/df22db55-52cd-47ca-8e8c-6ca9db62c9f1/volumes" Oct 30 01:38:46.821916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.821901 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" path="/var/lib/kubelet/pods/e9186ef0-4662-4afc-86a5-27d290426e41/volumes" Oct 30 01:38:46.892221 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.892184 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c56a37-9f9e-4635-96db-82383212e92c-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.892221 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.892221 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.892464 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.892236 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c56a37-9f9e-4635-96db-82383212e92c-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:46.892464 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:46.892248 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c56a37-9f9e-4635-96db-82383212e92c-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:47.556759 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.556728 5055 generic.go:356] "Generic (PLEG): container finished" podID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerID="303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1" exitCode=0 Oct 30 01:38:47.557225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.556812 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" event={"ID":"81a299b6-5136-4faa-99e6-4a9d0242869d","Type":"ContainerDied","Data":"303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1"} Oct 30 01:38:47.557292 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.557249 5055 scope.go:122] "RemoveContainer" containerID="303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1" Oct 30 01:38:47.558556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.558348 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" event={"ID":"09c56a37-9f9e-4635-96db-82383212e92c","Type":"ContainerDied","Data":"cd2dec5afc4b1e90e2a3322e05301e8bf5c55deb306bae6a1a353d36a4d843a7"} Oct 30 01:38:47.558556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.558399 5055 scope.go:122] "RemoveContainer" containerID="2e8ebf104e40759747dfff4bdef8cad508adf388cf0b804683a3bc86df4c46a9" Oct 30 01:38:47.558556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.558510 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp" Oct 30 01:38:47.558964 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.558933 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:47.559062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.559023 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:47.559123 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.559071 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:47.658119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.658091 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp"] Oct 30 01:38:47.666691 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:47.666636 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-l2tsp"] Oct 30 01:38:48.345865 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.345842 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:38:48.415575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.415548 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-data\") pod \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " Oct 30 01:38:48.415575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.415580 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-envoy\") pod \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " Oct 30 01:38:48.415796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.415612 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-podinfo\") pod \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " Oct 30 01:38:48.415796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.415652 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-kube-api-access-4g977\" (UniqueName: \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-kube-api-access-4g977\") pod \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " Oct 30 01:38:48.415796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.415733 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-socket\") pod \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " Oct 30 01:38:48.415796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.415779 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-credential-socket\") pod \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " Oct 30 01:38:48.415988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.415818 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-certs\") pod \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " Oct 30 01:38:48.415988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.415862 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-data" pod "086e7cd3-ff89-4558-8199-8fa2a6b7e326" (UID: "086e7cd3-ff89-4558-8199-8fa2a6b7e326"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:48.415988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.415930 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-token\" (UniqueName: \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-token\") pod \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " Oct 30 01:38:48.416137 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.416021 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istiod-ca-cert\") pod \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\" (UID: \"086e7cd3-ff89-4558-8199-8fa2a6b7e326\") " Oct 30 01:38:48.416137 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.416044 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-credential-socket" pod "086e7cd3-ff89-4558-8199-8fa2a6b7e326" (UID: "086e7cd3-ff89-4558-8199-8fa2a6b7e326"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:48.416137 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.416059 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-socket" pod "086e7cd3-ff89-4558-8199-8fa2a6b7e326" (UID: "086e7cd3-ff89-4558-8199-8fa2a6b7e326"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:48.416302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.416148 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-certs" pod "086e7cd3-ff89-4558-8199-8fa2a6b7e326" (UID: "086e7cd3-ff89-4558-8199-8fa2a6b7e326"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:48.416749 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.416422 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istiod-ca-cert" pod "086e7cd3-ff89-4558-8199-8fa2a6b7e326" (UID: "086e7cd3-ff89-4558-8199-8fa2a6b7e326"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:48.416901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.416879 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:48.416992 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.416909 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:48.416992 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.416925 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:48.416992 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.416939 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:48.416992 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.416952 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:48.418222 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.418201 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-envoy" pod "086e7cd3-ff89-4558-8199-8fa2a6b7e326" (UID: "086e7cd3-ff89-4558-8199-8fa2a6b7e326"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:48.418368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.418281 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-podinfo" pod "086e7cd3-ff89-4558-8199-8fa2a6b7e326" (UID: "086e7cd3-ff89-4558-8199-8fa2a6b7e326"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:48.418434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.418376 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-token" pod "086e7cd3-ff89-4558-8199-8fa2a6b7e326" (UID: "086e7cd3-ff89-4558-8199-8fa2a6b7e326"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:48.418434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.418401 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-kube-api-access-4g977" pod "086e7cd3-ff89-4558-8199-8fa2a6b7e326" (UID: "086e7cd3-ff89-4558-8199-8fa2a6b7e326"). InnerVolumeSpecName "kube-api-access-4g977". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:48.518171 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.518136 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:48.518171 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.518164 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:48.518171 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.518174 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g977\" (UniqueName: \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-kube-api-access-4g977\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:48.518418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.518184 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/086e7cd3-ff89-4558-8199-8fa2a6b7e326-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:38:48.601636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.601515 5055 generic.go:356] "Generic (PLEG): container finished" podID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerID="fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6" exitCode=0 Oct 30 01:38:48.601636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.601597 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" Oct 30 01:38:48.602170 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.601571 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" event={"ID":"086e7cd3-ff89-4558-8199-8fa2a6b7e326","Type":"ContainerDied","Data":"fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6"} Oct 30 01:38:48.602170 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.601816 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq" event={"ID":"086e7cd3-ff89-4558-8199-8fa2a6b7e326","Type":"ContainerDied","Data":"e17712be29528b864fc66f4bb133d71f340b5768545968b8e066f70a95097f6c"} Oct 30 01:38:48.602170 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.601845 5055 scope.go:122] "RemoveContainer" containerID="fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6" Oct 30 01:38:48.604598 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.604574 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" event={"ID":"81a299b6-5136-4faa-99e6-4a9d0242869d","Type":"ContainerStarted","Data":"1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f"} Oct 30 01:38:48.617359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.617318 5055 scope.go:122] "RemoveContainer" containerID="7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8" Oct 30 01:38:48.628562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.628546 5055 scope.go:122] "RemoveContainer" containerID="fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6" Oct 30 01:38:48.628791 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:38:48.628773 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6\": container with ID starting with fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6 not found: ID does not exist" containerID="fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6" Oct 30 01:38:48.628880 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.628802 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6"} err="failed to get container status \"fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6\": rpc error: code = NotFound desc = could not find container \"fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6\": container with ID starting with fd2ecacdca1a087cd722567b3628490e3aefd931e926bd0292de48f712e735d6 not found: ID does not exist" Oct 30 01:38:48.628880 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.628826 5055 scope.go:122] "RemoveContainer" containerID="7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8" Oct 30 01:38:48.629032 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:38:48.629013 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8\": container with ID starting with 7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8 not found: ID does not exist" containerID="7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8" Oct 30 01:38:48.629075 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.629043 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8"} err="failed to get container status \"7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8\": rpc error: code = NotFound desc = could not find container \"7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8\": container with ID starting with 7a33f5226ce7145b3242bddc2feb480307d1e50cfcf3eee8dde1b32a9949ece8 not found: ID does not exist" Oct 30 01:38:48.711775 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.711741 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq"] Oct 30 01:38:48.716007 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.715980 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f09934-openshift-default-748c86d47-27bdq"] Oct 30 01:38:48.822825 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.822789 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" path="/var/lib/kubelet/pods/086e7cd3-ff89-4558-8199-8fa2a6b7e326/volumes" Oct 30 01:38:48.823272 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.823256 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c56a37-9f9e-4635-96db-82383212e92c" path="/var/lib/kubelet/pods/09c56a37-9f9e-4635-96db-82383212e92c/volumes" Oct 30 01:38:48.850590 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.850569 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:38:48.850731 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.850713 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:38:48.850784 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:48.850756 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 01:38:49.228260 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:38:49.228190 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:38:49.851566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:49.851533 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:38:49.852164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:49.852135 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 01:38:50.851010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:50.850979 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:38:50.851188 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:50.851032 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 01:38:51.850690 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:51.850658 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:38:51.851062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:51.850704 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 01:38:52.850817 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:52.850743 5055 patch_prober.go:29] interesting pod/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 01:38:52.850817 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:52.850807 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 01:38:53.864862 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:53.864824 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:38:53.865397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:53.865213 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:38:53.866204 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:53.866186 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:38:55.235315 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.235272 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb"] Oct 30 01:38:55.239492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.239464 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6329c7b5-a9ae-4113-aceb-be3fbb71ab35" containerName="maas-api" Oct 30 01:38:55.239659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.239646 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6329c7b5-a9ae-4113-aceb-be3fbb71ab35" containerName="maas-api" Oct 30 01:38:55.239776 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.239766 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" containerName="istio-proxy" Oct 30 01:38:55.239856 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.239846 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" containerName="istio-proxy" Oct 30 01:38:55.239939 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.239930 5055 cpu_manager.go:414] "containerMap: removing container" podUID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" containerName="istio-proxy" Oct 30 01:38:55.240009 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.240000 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" containerName="istio-proxy" Oct 30 01:38:55.240084 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.240072 5055 cpu_manager.go:414] "containerMap: removing container" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" Oct 30 01:38:55.240158 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.240148 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" Oct 30 01:38:55.240235 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.240227 5055 cpu_manager.go:414] "containerMap: removing container" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" Oct 30 01:38:55.240308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.240298 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" Oct 30 01:38:55.240410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.240400 5055 cpu_manager.go:414] "containerMap: removing container" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" Oct 30 01:38:55.240494 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.240482 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" Oct 30 01:38:55.240617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.240606 5055 cpu_manager.go:414] "containerMap: removing container" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" Oct 30 01:38:55.240700 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.240689 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" Oct 30 01:38:55.240940 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.240927 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6329c7b5-a9ae-4113-aceb-be3fbb71ab35" containerName="maas-api" Oct 30 01:38:55.241037 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.241028 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9186ef0-4662-4afc-86a5-27d290426e41" containerName="istio-proxy" Oct 30 01:38:55.241111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.241103 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d4eff34-fdc7-40a5-b75d-9b8028822b94" containerName="istio-proxy" Oct 30 01:38:55.241177 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.241169 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" Oct 30 01:38:55.241248 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.241239 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="086e7cd3-ff89-4558-8199-8fa2a6b7e326" containerName="istio-proxy" Oct 30 01:38:55.241319 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.241311 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="df22db55-52cd-47ca-8e8c-6ca9db62c9f1" containerName="istio-proxy" Oct 30 01:38:55.241443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.241429 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c56a37-9f9e-4635-96db-82383212e92c" containerName="istio-proxy" Oct 30 01:38:55.257555 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.257516 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb"] Oct 30 01:38:55.257861 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.257838 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:38:55.280348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.275294 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f7bdcc-dockercfg-9mmwt\"" Oct 30 01:38:55.315995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.315971 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f7bdcc\"" Oct 30 01:38:55.342460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.342412 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd"] Oct 30 01:38:55.352339 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.352303 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:38:55.357858 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.357830 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4250e3b-3363-48ff-8e1b-f86d82f411cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb\" (UID: \"d4250e3b-3363-48ff-8e1b-f86d82f411cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:38:55.358135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.358111 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z67qj\" (UniqueName: \"kubernetes.io/projected/d4250e3b-3363-48ff-8e1b-f86d82f411cc-kube-api-access-z67qj\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb\" (UID: \"d4250e3b-3363-48ff-8e1b-f86d82f411cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:38:55.370148 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.370115 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd"] Oct 30 01:38:55.459266 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.459229 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7km\" (UniqueName: \"kubernetes.io/projected/84e365f7-3976-4260-9a0f-cb57aa05cbbe-kube-api-access-mw7km\") pod \"payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd\" (UID: \"84e365f7-3976-4260-9a0f-cb57aa05cbbe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:38:55.459588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.459556 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z67qj\" (UniqueName: \"kubernetes.io/projected/d4250e3b-3363-48ff-8e1b-f86d82f411cc-kube-api-access-z67qj\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb\" (UID: \"d4250e3b-3363-48ff-8e1b-f86d82f411cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:38:55.459805 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.459780 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e365f7-3976-4260-9a0f-cb57aa05cbbe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd\" (UID: \"84e365f7-3976-4260-9a0f-cb57aa05cbbe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:38:55.460069 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.459940 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4250e3b-3363-48ff-8e1b-f86d82f411cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb\" (UID: \"d4250e3b-3363-48ff-8e1b-f86d82f411cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:38:55.460878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.460854 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4250e3b-3363-48ff-8e1b-f86d82f411cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb\" (UID: \"d4250e3b-3363-48ff-8e1b-f86d82f411cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:38:55.480843 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.480809 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z67qj\" (UniqueName: \"kubernetes.io/projected/d4250e3b-3363-48ff-8e1b-f86d82f411cc-kube-api-access-z67qj\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb\" (UID: \"d4250e3b-3363-48ff-8e1b-f86d82f411cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:38:55.561244 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.561135 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7km\" (UniqueName: \"kubernetes.io/projected/84e365f7-3976-4260-9a0f-cb57aa05cbbe-kube-api-access-mw7km\") pod \"payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd\" (UID: \"84e365f7-3976-4260-9a0f-cb57aa05cbbe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:38:55.561443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.561307 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e365f7-3976-4260-9a0f-cb57aa05cbbe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd\" (UID: \"84e365f7-3976-4260-9a0f-cb57aa05cbbe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:38:55.562461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.562433 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e365f7-3976-4260-9a0f-cb57aa05cbbe-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd\" (UID: \"84e365f7-3976-4260-9a0f-cb57aa05cbbe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:38:55.583547 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.583366 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7km\" (UniqueName: \"kubernetes.io/projected/84e365f7-3976-4260-9a0f-cb57aa05cbbe-kube-api-access-mw7km\") pod \"payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd\" (UID: \"84e365f7-3976-4260-9a0f-cb57aa05cbbe\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:38:55.648813 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.648779 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:38:55.697906 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.697883 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:38:55.970213 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:38:55.970178 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4250e3b_3363_48ff_8e1b_f86d82f411cc.slice/crio-4effeb729092954364e48c74c52607e1bbb1aaedde54466712509aeb162e7475 WatchSource:0}: Error finding container 4effeb729092954364e48c74c52607e1bbb1aaedde54466712509aeb162e7475: Status 404 returned error can't find the container with id 4effeb729092954364e48c74c52607e1bbb1aaedde54466712509aeb162e7475 Oct 30 01:38:55.971990 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.971963 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb"] Oct 30 01:38:55.972730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:55.972715 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:38:56.027522 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:38:56.027494 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e365f7_3976_4260_9a0f_cb57aa05cbbe.slice/crio-ba21df2f81a0eeebcc414c5a7c24a05a9e6dbeb4bc31dc68cdf068c0c8e99f24 WatchSource:0}: Error finding container ba21df2f81a0eeebcc414c5a7c24a05a9e6dbeb4bc31dc68cdf068c0c8e99f24: Status 404 returned error can't find the container with id ba21df2f81a0eeebcc414c5a7c24a05a9e6dbeb4bc31dc68cdf068c0c8e99f24 Oct 30 01:38:56.029779 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.029748 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd"] Oct 30 01:38:56.458118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.458086 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6"] Oct 30 01:38:56.463185 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.463164 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.480105 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.480076 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-fc51e9-openshift-default-dockercfg-fwqpg\"" Oct 30 01:38:56.498344 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.498022 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6"] Oct 30 01:38:56.580355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.577500 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-certs\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.580355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.577673 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-credential-socket\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.580355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.577878 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-socket\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.580355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.578027 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-data\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.580355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.578065 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-podinfo\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.580355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.578183 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041906cb-d475-4df3-9a26-b3b422fe5eae-istiod-ca-cert\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.580355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.578422 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j982p\" (UniqueName: \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-kube-api-access-j982p\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.580355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.578479 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-token\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.580355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.578507 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-envoy\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.685384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.680113 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j982p\" (UniqueName: \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-kube-api-access-j982p\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.685384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.680192 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-token\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.685384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.680228 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-envoy\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.685384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.680318 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-certs\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.685384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.680407 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-credential-socket\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.685384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.680478 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-socket\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.685384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.680512 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-data\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.685384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.680548 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-podinfo\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.685384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.680630 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041906cb-d475-4df3-9a26-b3b422fe5eae-istiod-ca-cert\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.685384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.681503 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041906cb-d475-4df3-9a26-b3b422fe5eae-istiod-ca-cert\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.691416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.688882 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-socket\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.691416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.688911 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-certs\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.691416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.689162 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-data\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.691416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.689215 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-credential-socket\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.706351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.698943 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-podinfo\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.734725 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.734586 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" event={"ID":"84e365f7-3976-4260-9a0f-cb57aa05cbbe","Type":"ContainerStarted","Data":"ba21df2f81a0eeebcc414c5a7c24a05a9e6dbeb4bc31dc68cdf068c0c8e99f24"} Oct 30 01:38:56.755425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.746182 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" event={"ID":"d4250e3b-3363-48ff-8e1b-f86d82f411cc","Type":"ContainerStarted","Data":"4effeb729092954364e48c74c52607e1bbb1aaedde54466712509aeb162e7475"} Oct 30 01:38:56.755425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.755172 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-token\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.768436 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.763800 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j982p\" (UniqueName: \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-kube-api-access-j982p\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.775393 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.771669 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-envoy\") pod \"e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:56.802232 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:56.801564 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:57.244321 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:38:57.244287 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod041906cb_d475_4df3_9a26_b3b422fe5eae.slice/crio-15df5cb26748105e668d7ae43719b9ce3afe0660dc1ca9759d4b785b3f469215 WatchSource:0}: Error finding container 15df5cb26748105e668d7ae43719b9ce3afe0660dc1ca9759d4b785b3f469215: Status 404 returned error can't find the container with id 15df5cb26748105e668d7ae43719b9ce3afe0660dc1ca9759d4b785b3f469215 Oct 30 01:38:57.245006 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:57.244983 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6"] Oct 30 01:38:57.248963 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:57.248914 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:57.249320 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:57.249001 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:57.249320 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:57.249053 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:57.759940 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:57.759871 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" event={"ID":"041906cb-d475-4df3-9a26-b3b422fe5eae","Type":"ContainerStarted","Data":"2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567"} Oct 30 01:38:57.759940 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:57.759937 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" event={"ID":"041906cb-d475-4df3-9a26-b3b422fe5eae","Type":"ContainerStarted","Data":"15df5cb26748105e668d7ae43719b9ce3afe0660dc1ca9759d4b785b3f469215"} Oct 30 01:38:57.804615 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:57.804588 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:57.833422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:57.831855 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" podStartSLOduration=1.831835401 podStartE2EDuration="1.831835401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:57.828812948 +0000 UTC m=+1305.149769999" watchObservedRunningTime="2026-10-30 01:38:57.831835401 +0000 UTC m=+1305.152792442" Oct 30 01:38:58.788414 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.788375 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" event={"ID":"d4250e3b-3363-48ff-8e1b-f86d82f411cc","Type":"ContainerStarted","Data":"094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd"} Oct 30 01:38:58.789906 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.789884 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:38:58.792505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.792474 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" event={"ID":"84e365f7-3976-4260-9a0f-cb57aa05cbbe","Type":"ContainerStarted","Data":"ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a"} Oct 30 01:38:58.793060 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.793038 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:38:58.809471 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.809448 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:58.860008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.859976 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn"] Oct 30 01:38:58.864868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.864809 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" podStartSLOduration=1.634520632 podStartE2EDuration="3.864790169s" totalImagesPullingTime="2.230269537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:58.86291947 +0000 UTC m=+1306.183876511" watchObservedRunningTime="2026-10-30 01:38:58.864790169 +0000 UTC m=+1306.185747211" Oct 30 01:38:58.893147 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.893121 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn"] Oct 30 01:38:58.893521 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.893441 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:58.913691 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.913670 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d8eb06-openshift-default-dockercfg-lfrvq\"" Oct 30 01:38:58.915016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.914971 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" podStartSLOduration=1.697304893 podStartE2EDuration="3.914954626s" totalImagesPullingTime="2.217649733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:58.914021222 +0000 UTC m=+1306.234978267" watchObservedRunningTime="2026-10-30 01:38:58.914954626 +0000 UTC m=+1306.235911667" Oct 30 01:38:58.919201 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.919179 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-credential-socket\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:58.919510 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.919495 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk9vb\" (UniqueName: \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-kube-api-access-sk9vb\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:58.919669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.919653 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-data\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:58.919878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.919865 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/316b9154-8e84-4e72-9250-d67591d60ed5-istiod-ca-cert\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:58.920055 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.920042 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/316b9154-8e84-4e72-9250-d67591d60ed5-istio-podinfo\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:58.920192 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.920180 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-socket\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:58.921156 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.921137 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-envoy\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:58.921379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.921364 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-istio-token\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:58.921482 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:58.921469 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-certs\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.022850 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.022814 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-istio-token\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.022850 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.022853 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-certs\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.023128 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.022902 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-credential-socket\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.023128 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.022961 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk9vb\" (UniqueName: \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-kube-api-access-sk9vb\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.023128 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.022985 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-data\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.023128 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.023098 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/316b9154-8e84-4e72-9250-d67591d60ed5-istiod-ca-cert\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.023566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.023190 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/316b9154-8e84-4e72-9250-d67591d60ed5-istio-podinfo\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.023566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.023243 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-socket\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.023566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.023268 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-envoy\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.023566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.023409 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-certs\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.023566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.023506 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-credential-socket\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.023937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.023809 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-socket\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.024158 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.024025 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/316b9154-8e84-4e72-9250-d67591d60ed5-istiod-ca-cert\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.024158 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.024042 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-data\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.026538 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.026514 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/316b9154-8e84-4e72-9250-d67591d60ed5-istio-podinfo\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.028045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.028024 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-envoy\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.045615 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.045547 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-istio-token\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.049921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.049893 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk9vb\" (UniqueName: \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-kube-api-access-sk9vb\") pod \"e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.225351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.224316 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:38:59.652222 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.652137 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn"] Oct 30 01:38:59.668071 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:38:59.668035 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316b9154_8e84_4e72_9250_d67591d60ed5.slice/crio-ab0841155c51e750b088c86320c3f182a33ea1ee2110281d1289da6a0cd577cc WatchSource:0}: Error finding container ab0841155c51e750b088c86320c3f182a33ea1ee2110281d1289da6a0cd577cc: Status 404 returned error can't find the container with id ab0841155c51e750b088c86320c3f182a33ea1ee2110281d1289da6a0cd577cc Oct 30 01:38:59.671313 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.671277 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:59.671424 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.671366 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:59.671424 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.671399 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:38:59.881851 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.881817 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" event={"ID":"316b9154-8e84-4e72-9250-d67591d60ed5","Type":"ContainerStarted","Data":"ab0841155c51e750b088c86320c3f182a33ea1ee2110281d1289da6a0cd577cc"} Oct 30 01:38:59.882431 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.881880 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:38:59.883597 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:38:59.883580 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:39:00.890273 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:00.890236 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" event={"ID":"316b9154-8e84-4e72-9250-d67591d60ed5","Type":"ContainerStarted","Data":"12ccbc63fbc85733eb094094f349dd5c2619e2b18fc7aa336a8e60c5f4d93958"} Oct 30 01:39:00.949731 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:00.949680 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" podStartSLOduration=2.949665187 podStartE2EDuration="2.949665187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:00.94614052 +0000 UTC m=+1308.267097561" watchObservedRunningTime="2026-10-30 01:39:00.949665187 +0000 UTC m=+1308.270622227" Oct 30 01:39:01.225755 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:01.225713 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:39:01.232941 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:01.232915 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:39:01.898972 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:01.898944 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:39:01.899925 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:01.899902 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:39:09.885798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:09.885762 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:39:10.893099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:10.893070 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:39:13.883621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:13.883590 5055 scope.go:122] "RemoveContainer" containerID="b03e45fb17840c69433ff7b293e259a9947935202de696e3b50d07021720f845" Oct 30 01:39:13.893955 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:13.893934 5055 scope.go:122] "RemoveContainer" containerID="3cc3494dfb91177b5f35acd69f063b4151f3c3c397a5138b638f8661c7d0a668" Oct 30 01:39:13.903501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:13.903473 5055 scope.go:122] "RemoveContainer" containerID="90e89f86bb2070acf4862bfbc35c9489f7e3718f975a4e4e83b0c28305416c71" Oct 30 01:39:19.221079 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:39:19.221030 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:39:21.800851 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.800813 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g"] Oct 30 01:39:21.805425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.805405 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.837906 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.837879 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-fc51e9-serving-cert\"" Oct 30 01:39:21.856077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.856040 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g"] Oct 30 01:39:21.892760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.892688 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-metrics-certs\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.893120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.893096 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.893401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.893378 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4553d4-7bf8-4fdc-b6a9-2036c1300826-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.893710 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.893676 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxqks\" (UniqueName: \"kubernetes.io/projected/db4553d4-7bf8-4fdc-b6a9-2036c1300826-kube-api-access-rxqks\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.995109 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.995074 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxqks\" (UniqueName: \"kubernetes.io/projected/db4553d4-7bf8-4fdc-b6a9-2036c1300826-kube-api-access-rxqks\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.995306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.995160 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-metrics-certs\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.995306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.995239 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.995492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.995379 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4553d4-7bf8-4fdc-b6a9-2036c1300826-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.996069 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.995993 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4553d4-7bf8-4fdc-b6a9-2036c1300826-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.997863 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.997838 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-metrics-certs\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:21.997987 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:21.997879 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-maas-api-tls\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:22.015514 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:22.015486 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxqks\" (UniqueName: \"kubernetes.io/projected/db4553d4-7bf8-4fdc-b6a9-2036c1300826-kube-api-access-rxqks\") pod \"maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:22.130411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:22.126938 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:22.489884 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:39:22.489855 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4553d4_7bf8_4fdc_b6a9_2036c1300826.slice/crio-ef1e264aef454d93f198a1a8ef159ab10339fa9962b946df17e451cdba57151f WatchSource:0}: Error finding container ef1e264aef454d93f198a1a8ef159ab10339fa9962b946df17e451cdba57151f: Status 404 returned error can't find the container with id ef1e264aef454d93f198a1a8ef159ab10339fa9962b946df17e451cdba57151f Oct 30 01:39:22.491990 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:22.491923 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g"] Oct 30 01:39:23.117754 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:23.116532 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" event={"ID":"db4553d4-7bf8-4fdc-b6a9-2036c1300826","Type":"ContainerStarted","Data":"728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a"} Oct 30 01:39:23.117754 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:23.116598 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" event={"ID":"db4553d4-7bf8-4fdc-b6a9-2036c1300826","Type":"ContainerStarted","Data":"ef1e264aef454d93f198a1a8ef159ab10339fa9962b946df17e451cdba57151f"} Oct 30 01:39:23.117754 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:23.117685 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:23.159298 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:23.159182 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" podStartSLOduration=1.767342223 podStartE2EDuration="2.159161905s" totalImagesPullingTime="391.819682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:23.158165816 +0000 UTC m=+1330.479122856" watchObservedRunningTime="2026-10-30 01:39:23.159161905 +0000 UTC m=+1330.480118946" Oct 30 01:39:25.884534 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:25.884485 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6"] Oct 30 01:39:25.889638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:25.889605 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:25.916536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:25.916491 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d8eb06-serving-cert\"" Oct 30 01:39:25.920272 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:25.920243 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6"] Oct 30 01:39:25.976005 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:25.975964 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25014618-d292-485b-85d7-80a086f2eb0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:25.976217 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:25.976132 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:25.976217 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:25.976181 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-metrics-certs\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:25.976346 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:25.976257 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtbzf\" (UniqueName: \"kubernetes.io/projected/25014618-d292-485b-85d7-80a086f2eb0f-kube-api-access-jtbzf\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:26.079082 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:26.079042 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtbzf\" (UniqueName: \"kubernetes.io/projected/25014618-d292-485b-85d7-80a086f2eb0f-kube-api-access-jtbzf\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:26.079298 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:26.079275 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25014618-d292-485b-85d7-80a086f2eb0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:26.079426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:26.079405 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:26.079514 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:26.079454 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-metrics-certs\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:26.080259 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:26.080224 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25014618-d292-485b-85d7-80a086f2eb0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:26.084707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:26.084680 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:26.118945 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:26.118913 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-metrics-certs\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:26.142139 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:26.127203 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtbzf\" (UniqueName: \"kubernetes.io/projected/25014618-d292-485b-85d7-80a086f2eb0f-kube-api-access-jtbzf\") pod \"maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:26.208426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:26.208391 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:26.516150 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:39:26.516115 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25014618_d292_485b_85d7_80a086f2eb0f.slice/crio-983a3fe99a884b6a3497a00cf7c92929359ebec55424858ef60ad97ca03f55c6 WatchSource:0}: Error finding container 983a3fe99a884b6a3497a00cf7c92929359ebec55424858ef60ad97ca03f55c6: Status 404 returned error can't find the container with id 983a3fe99a884b6a3497a00cf7c92929359ebec55424858ef60ad97ca03f55c6 Oct 30 01:39:26.517274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:26.517238 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6"] Oct 30 01:39:27.165063 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:27.163785 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" event={"ID":"25014618-d292-485b-85d7-80a086f2eb0f","Type":"ContainerStarted","Data":"e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad"} Oct 30 01:39:27.165063 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:27.163853 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" event={"ID":"25014618-d292-485b-85d7-80a086f2eb0f","Type":"ContainerStarted","Data":"983a3fe99a884b6a3497a00cf7c92929359ebec55424858ef60ad97ca03f55c6"} Oct 30 01:39:27.165063 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:27.164929 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:27.207995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:27.207941 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" podStartSLOduration=1.822371229 podStartE2EDuration="2.207926936s" totalImagesPullingTime="385.555707ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:27.204526862 +0000 UTC m=+1334.525483899" watchObservedRunningTime="2026-10-30 01:39:27.207926936 +0000 UTC m=+1334.528884012" Oct 30 01:39:30.134753 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:30.134722 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:39:34.187774 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:34.187747 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:39:49.316351 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:39:49.316301 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:39:50.620358 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:50.602871 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg"] Oct 30 01:39:50.620358 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:50.603129 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" containerID="cri-o://b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689" gracePeriod=30 Oct 30 01:39:56.042794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.042766 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:39:56.159146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.159094 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-certs\") pod \"f7f0697a-4001-46eb-8bf6-83200b8d0062\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " Oct 30 01:39:56.159592 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.159564 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-kube-api-access-2gmtp\" (UniqueName: \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-kube-api-access-2gmtp\") pod \"f7f0697a-4001-46eb-8bf6-83200b8d0062\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " Oct 30 01:39:56.159592 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.159570 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-certs" pod "f7f0697a-4001-46eb-8bf6-83200b8d0062" (UID: "f7f0697a-4001-46eb-8bf6-83200b8d0062"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:56.159745 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.159623 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-socket\") pod \"f7f0697a-4001-46eb-8bf6-83200b8d0062\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " Oct 30 01:39:56.159745 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.159711 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-credential-socket\") pod \"f7f0697a-4001-46eb-8bf6-83200b8d0062\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " Oct 30 01:39:56.159852 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.159755 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-token\" (UniqueName: \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-token\") pod \"f7f0697a-4001-46eb-8bf6-83200b8d0062\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " Oct 30 01:39:56.159852 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.159813 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-envoy\") pod \"f7f0697a-4001-46eb-8bf6-83200b8d0062\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " Oct 30 01:39:56.159954 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.159879 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-data\") pod \"f7f0697a-4001-46eb-8bf6-83200b8d0062\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " Oct 30 01:39:56.159954 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.159924 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f0697a-4001-46eb-8bf6-83200b8d0062-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f0697a-4001-46eb-8bf6-83200b8d0062-istiod-ca-cert\") pod \"f7f0697a-4001-46eb-8bf6-83200b8d0062\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " Oct 30 01:39:56.160089 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.160073 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-podinfo\") pod \"f7f0697a-4001-46eb-8bf6-83200b8d0062\" (UID: \"f7f0697a-4001-46eb-8bf6-83200b8d0062\") " Oct 30 01:39:56.160802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.160777 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.162352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.162053 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-socket" pod "f7f0697a-4001-46eb-8bf6-83200b8d0062" (UID: "f7f0697a-4001-46eb-8bf6-83200b8d0062"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:56.162731 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.162620 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f0697a-4001-46eb-8bf6-83200b8d0062-istiod-ca-cert" pod "f7f0697a-4001-46eb-8bf6-83200b8d0062" (UID: "f7f0697a-4001-46eb-8bf6-83200b8d0062"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:56.162731 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.162702 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-credential-socket" pod "f7f0697a-4001-46eb-8bf6-83200b8d0062" (UID: "f7f0697a-4001-46eb-8bf6-83200b8d0062"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:56.163286 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.163264 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-data" pod "f7f0697a-4001-46eb-8bf6-83200b8d0062" (UID: "f7f0697a-4001-46eb-8bf6-83200b8d0062"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:56.163834 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.163807 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-kube-api-access-2gmtp" pod "f7f0697a-4001-46eb-8bf6-83200b8d0062" (UID: "f7f0697a-4001-46eb-8bf6-83200b8d0062"). InnerVolumeSpecName "kube-api-access-2gmtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:56.177377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.177060 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-token" pod "f7f0697a-4001-46eb-8bf6-83200b8d0062" (UID: "f7f0697a-4001-46eb-8bf6-83200b8d0062"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:56.177377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.177174 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-envoy" pod "f7f0697a-4001-46eb-8bf6-83200b8d0062" (UID: "f7f0697a-4001-46eb-8bf6-83200b8d0062"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:56.178920 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.178894 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-podinfo" pod "f7f0697a-4001-46eb-8bf6-83200b8d0062" (UID: "f7f0697a-4001-46eb-8bf6-83200b8d0062"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:39:56.263081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.262974 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.263081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.263017 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f0697a-4001-46eb-8bf6-83200b8d0062-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.263081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.263032 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.263081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.263046 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gmtp\" (UniqueName: \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-kube-api-access-2gmtp\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.263081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.263062 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.263081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.263078 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.263541 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.263092 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.263541 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.263106 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f0697a-4001-46eb-8bf6-83200b8d0062-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:39:56.542659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.542570 5055 generic.go:356] "Generic (PLEG): container finished" podID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerID="b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689" exitCode=0 Oct 30 01:39:56.542659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.542654 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" Oct 30 01:39:56.542874 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.542656 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" event={"ID":"f7f0697a-4001-46eb-8bf6-83200b8d0062","Type":"ContainerDied","Data":"b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689"} Oct 30 01:39:56.542874 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.542777 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg" event={"ID":"f7f0697a-4001-46eb-8bf6-83200b8d0062","Type":"ContainerDied","Data":"596db85e61879c1398635c279d5a27f04b0cb568350bb69ff300650d97b5871b"} Oct 30 01:39:56.542874 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.542801 5055 scope.go:122] "RemoveContainer" containerID="b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689" Oct 30 01:39:56.554277 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.554261 5055 scope.go:122] "RemoveContainer" containerID="355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565" Oct 30 01:39:56.565032 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.565010 5055 scope.go:122] "RemoveContainer" containerID="b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689" Oct 30 01:39:56.566023 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:39:56.565999 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689\": container with ID starting with b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689 not found: ID does not exist" containerID="b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689" Oct 30 01:39:56.566113 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.566036 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689"} err="failed to get container status \"b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689\": rpc error: code = NotFound desc = could not find container \"b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689\": container with ID starting with b5fb900d1645e96939364380d9b44fcffe32abf7dc76abd8b642a67325638689 not found: ID does not exist" Oct 30 01:39:56.566113 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.566063 5055 scope.go:122] "RemoveContainer" containerID="355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565" Oct 30 01:39:56.566438 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:39:56.566413 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565\": container with ID starting with 355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565 not found: ID does not exist" containerID="355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565" Oct 30 01:39:56.566527 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.566446 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565"} err="failed to get container status \"355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565\": rpc error: code = NotFound desc = could not find container \"355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565\": container with ID starting with 355dbdbe455d59aa6437023d165665f96b6cbb669d46188cfa340a1c06308565 not found: ID does not exist" Oct 30 01:39:56.630703 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.606628 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg"] Oct 30 01:39:56.630703 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.617469 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18560b-openshift-default-6f69c45855-s5hsg"] Oct 30 01:39:56.825479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:39:56.825391 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" path="/var/lib/kubelet/pods/f7f0697a-4001-46eb-8bf6-83200b8d0062/volumes" Oct 30 01:40:00.027414 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.025769 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": read tcp 10.132.0.2:40812->10.132.0.112:15021: read: connection reset by peer" start-of-body= Oct 30 01:40:00.027414 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.025827 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": read tcp 10.132.0.2:40812->10.132.0.112:15021: read: connection reset by peer" Oct 30 01:40:00.597052 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.596974 5055 generic.go:356] "Generic (PLEG): container finished" podID="316b9154-8e84-4e72-9250-d67591d60ed5" containerID="12ccbc63fbc85733eb094094f349dd5c2619e2b18fc7aa336a8e60c5f4d93958" exitCode=0 Oct 30 01:40:00.597260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.597052 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" event={"ID":"316b9154-8e84-4e72-9250-d67591d60ed5","Type":"ContainerDied","Data":"12ccbc63fbc85733eb094094f349dd5c2619e2b18fc7aa336a8e60c5f4d93958"} Oct 30 01:40:00.597628 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.597608 5055 scope.go:122] "RemoveContainer" containerID="12ccbc63fbc85733eb094094f349dd5c2619e2b18fc7aa336a8e60c5f4d93958" Oct 30 01:40:00.599127 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.599100 5055 generic.go:356] "Generic (PLEG): container finished" podID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerID="2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567" exitCode=0 Oct 30 01:40:00.599205 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.599184 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" event={"ID":"041906cb-d475-4df3-9a26-b3b422fe5eae","Type":"ContainerDied","Data":"2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567"} Oct 30 01:40:00.599273 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.599250 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:40:00.599354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.599321 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:40:00.599405 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.599382 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:40:00.599628 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.599612 5055 scope.go:122] "RemoveContainer" containerID="2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567" Oct 30 01:40:00.601382 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.601347 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:40:00.601460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.601432 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:40:00.601514 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:00.601479 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:40:01.648434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:01.648393 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" event={"ID":"316b9154-8e84-4e72-9250-d67591d60ed5","Type":"ContainerStarted","Data":"1a0b18a2b67c3e4b9860fd323e0720965d8c20620d866987371a3e6ecb1672e3"} Oct 30 01:40:01.654356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:01.651462 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" event={"ID":"041906cb-d475-4df3-9a26-b3b422fe5eae","Type":"ContainerStarted","Data":"bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8"} Oct 30 01:40:01.810352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:01.806721 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:40:01.810352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:01.806854 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 01:40:01.810352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:01.806912 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 01:40:01.900412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:01.899542 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:40:02.226437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:02.226388 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:40:02.227600 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:02.227572 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:40:02.227734 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:02.227631 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 01:40:02.808348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:02.804685 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 01:40:02.808348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:02.804743 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 01:40:03.227379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:03.225460 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:40:03.227379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:03.225517 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 01:40:03.816461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:03.816430 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 01:40:03.817081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:03.817035 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 01:40:04.225705 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:04.225668 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:40:04.226018 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:04.225988 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 01:40:04.815741 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:04.815707 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 01:40:04.816036 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:04.816009 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 01:40:05.230351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:05.225792 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:40:05.230351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:05.225851 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 01:40:05.805775 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:05.805737 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 01:40:05.806074 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:05.806046 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 01:40:06.259937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:06.259522 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:40:06.259937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:06.259581 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 01:40:06.803710 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:06.803673 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 01:40:06.804023 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:06.803990 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 01:40:07.229349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:07.228794 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 30 01:40:07.229349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:07.228854 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 30 01:40:07.803678 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:07.803623 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 01:40:07.804179 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:07.803702 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 01:40:08.329899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:08.329852 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:40:08.334537 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:08.334512 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:40:08.861798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:08.861770 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:40:08.878578 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:08.878556 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:40:08.918370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:08.914525 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:40:10.021488 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:10.021452 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8qtbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:40:10.022141 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:40:10.022109 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:40:19.219517 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:40:19.219479 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:40:49.219064 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:40:49.218979 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:41:19.218990 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:19.218958 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:41:19.892447 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:19.892406 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g"] Oct 30 01:41:19.892674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:19.892616 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" podUID="db4553d4-7bf8-4fdc-b6a9-2036c1300826" containerName="maas-api" containerID="cri-o://728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a" gracePeriod=30 Oct 30 01:41:20.253094 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.253066 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:41:20.337315 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.337273 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9"] Oct 30 01:41:20.337654 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.337591 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" podUID="1736cd25-3aa2-4ea0-8557-8f58096599df" containerName="maas-api" containerID="cri-o://9f01bee13c98ad5537115c4aba03f33168d3a5a0293294b15fca9a5963b91bb5" gracePeriod=30 Oct 30 01:41:20.344338 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.343370 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4553d4-7bf8-4fdc-b6a9-2036c1300826-kube-api-access-rxqks\" (UniqueName: \"kubernetes.io/projected/db4553d4-7bf8-4fdc-b6a9-2036c1300826-kube-api-access-rxqks\") pod \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " Oct 30 01:41:20.344338 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.343579 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-maas-api-tls\") pod \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " Oct 30 01:41:20.344338 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.343624 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-metrics-certs\") pod \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " Oct 30 01:41:20.344338 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.343754 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4553d4-7bf8-4fdc-b6a9-2036c1300826-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4553d4-7bf8-4fdc-b6a9-2036c1300826-openshift-service-ca\") pod \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\" (UID: \"db4553d4-7bf8-4fdc-b6a9-2036c1300826\") " Oct 30 01:41:20.345722 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.345432 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4553d4-7bf8-4fdc-b6a9-2036c1300826-openshift-service-ca" pod "db4553d4-7bf8-4fdc-b6a9-2036c1300826" (UID: "db4553d4-7bf8-4fdc-b6a9-2036c1300826"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:20.351014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.350992 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-maas-api-tls" pod "db4553d4-7bf8-4fdc-b6a9-2036c1300826" (UID: "db4553d4-7bf8-4fdc-b6a9-2036c1300826"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:20.366055 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.366010 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4553d4-7bf8-4fdc-b6a9-2036c1300826-kube-api-access-rxqks" pod "db4553d4-7bf8-4fdc-b6a9-2036c1300826" (UID: "db4553d4-7bf8-4fdc-b6a9-2036c1300826"). InnerVolumeSpecName "kube-api-access-rxqks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:20.367229 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.367188 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-metrics-certs" pod "db4553d4-7bf8-4fdc-b6a9-2036c1300826" (UID: "db4553d4-7bf8-4fdc-b6a9-2036c1300826"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:20.447780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.447747 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4553d4-7bf8-4fdc-b6a9-2036c1300826-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.447780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.447780 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxqks\" (UniqueName: \"kubernetes.io/projected/db4553d4-7bf8-4fdc-b6a9-2036c1300826-kube-api-access-rxqks\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.448013 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.447795 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.448013 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.447809 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db4553d4-7bf8-4fdc-b6a9-2036c1300826-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.587567 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.587461 5055 generic.go:356] "Generic (PLEG): container finished" podID="1736cd25-3aa2-4ea0-8557-8f58096599df" containerID="9f01bee13c98ad5537115c4aba03f33168d3a5a0293294b15fca9a5963b91bb5" exitCode=0 Oct 30 01:41:20.587990 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.587782 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" event={"ID":"1736cd25-3aa2-4ea0-8557-8f58096599df","Type":"ContainerDied","Data":"9f01bee13c98ad5537115c4aba03f33168d3a5a0293294b15fca9a5963b91bb5"} Oct 30 01:41:20.589715 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.589692 5055 generic.go:356] "Generic (PLEG): container finished" podID="db4553d4-7bf8-4fdc-b6a9-2036c1300826" containerID="728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a" exitCode=0 Oct 30 01:41:20.589847 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.589752 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" event={"ID":"db4553d4-7bf8-4fdc-b6a9-2036c1300826","Type":"ContainerDied","Data":"728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a"} Oct 30 01:41:20.589847 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.589791 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" event={"ID":"db4553d4-7bf8-4fdc-b6a9-2036c1300826","Type":"ContainerDied","Data":"ef1e264aef454d93f198a1a8ef159ab10339fa9962b946df17e451cdba57151f"} Oct 30 01:41:20.589847 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.589818 5055 scope.go:122] "RemoveContainer" containerID="728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a" Oct 30 01:41:20.590783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.590020 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" Oct 30 01:41:20.616513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.614550 5055 scope.go:122] "RemoveContainer" containerID="728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a" Oct 30 01:41:20.616513 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:20.616047 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a\": container with ID starting with 728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a not found: ID does not exist" containerID="728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a" Oct 30 01:41:20.616513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.616087 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a"} err="failed to get container status \"728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a\": rpc error: code = NotFound desc = could not find container \"728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a\": container with ID starting with 728bb4e19954aaf812f987a5717774fda294f3a64bf5d1c7bbc5af471a1e295a not found: ID does not exist" Oct 30 01:41:20.661949 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.661837 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g"] Oct 30 01:41:20.666344 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.666262 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g"] Oct 30 01:41:20.748672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.748645 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:41:20.826913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.826870 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4553d4-7bf8-4fdc-b6a9-2036c1300826" path="/var/lib/kubelet/pods/db4553d4-7bf8-4fdc-b6a9-2036c1300826/volumes" Oct 30 01:41:20.865425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.864730 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1736cd25-3aa2-4ea0-8557-8f58096599df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1736cd25-3aa2-4ea0-8557-8f58096599df-openshift-service-ca\") pod \"1736cd25-3aa2-4ea0-8557-8f58096599df\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " Oct 30 01:41:20.865425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.864790 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1736cd25-3aa2-4ea0-8557-8f58096599df-kube-api-access-tr7fw\" (UniqueName: \"kubernetes.io/projected/1736cd25-3aa2-4ea0-8557-8f58096599df-kube-api-access-tr7fw\") pod \"1736cd25-3aa2-4ea0-8557-8f58096599df\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " Oct 30 01:41:20.865425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.864823 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-metrics-certs\") pod \"1736cd25-3aa2-4ea0-8557-8f58096599df\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " Oct 30 01:41:20.865425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.864852 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-maas-api-tls\") pod \"1736cd25-3aa2-4ea0-8557-8f58096599df\" (UID: \"1736cd25-3aa2-4ea0-8557-8f58096599df\") " Oct 30 01:41:20.866256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.866162 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1736cd25-3aa2-4ea0-8557-8f58096599df-openshift-service-ca" pod "1736cd25-3aa2-4ea0-8557-8f58096599df" (UID: "1736cd25-3aa2-4ea0-8557-8f58096599df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:20.868400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.868374 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1736cd25-3aa2-4ea0-8557-8f58096599df-kube-api-access-tr7fw" pod "1736cd25-3aa2-4ea0-8557-8f58096599df" (UID: "1736cd25-3aa2-4ea0-8557-8f58096599df"). InnerVolumeSpecName "kube-api-access-tr7fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:20.869262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.869237 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-metrics-certs" pod "1736cd25-3aa2-4ea0-8557-8f58096599df" (UID: "1736cd25-3aa2-4ea0-8557-8f58096599df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:20.869591 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.869570 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-maas-api-tls" pod "1736cd25-3aa2-4ea0-8557-8f58096599df" (UID: "1736cd25-3aa2-4ea0-8557-8f58096599df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:20.993353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.978552 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1736cd25-3aa2-4ea0-8557-8f58096599df-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.993353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.978587 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr7fw\" (UniqueName: \"kubernetes.io/projected/1736cd25-3aa2-4ea0-8557-8f58096599df-kube-api-access-tr7fw\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.993353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.978601 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:20.993353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:20.978612 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1736cd25-3aa2-4ea0-8557-8f58096599df-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:21.598599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:21.598565 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" event={"ID":"1736cd25-3aa2-4ea0-8557-8f58096599df","Type":"ContainerDied","Data":"5214a144bcd1ff12a2c8dd785fd060961cf1dc3361385086599667fe5d39c9f4"} Oct 30 01:41:21.599110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:21.598620 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9" Oct 30 01:41:21.599110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:21.598634 5055 scope.go:122] "RemoveContainer" containerID="9f01bee13c98ad5537115c4aba03f33168d3a5a0293294b15fca9a5963b91bb5" Oct 30 01:41:21.690549 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:21.690512 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9"] Oct 30 01:41:21.706660 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:21.706626 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-d9ede0-566774c9c9-8r2l9"] Oct 30 01:41:22.822940 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:22.822900 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1736cd25-3aa2-4ea0-8557-8f58096599df" path="/var/lib/kubelet/pods/1736cd25-3aa2-4ea0-8557-8f58096599df/volumes" Oct 30 01:41:23.134777 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:23.134687 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-fc51e9-5f596dcb6d-tkf2g" podUID="db4553d4-7bf8-4fdc-b6a9-2036c1300826" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.114:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:41:28.353678 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:28.353565 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6"] Oct 30 01:41:28.354158 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:28.353884 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" containerID="cri-o://bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8" gracePeriod=30 Oct 30 01:41:29.015304 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:29.015267 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255"] Oct 30 01:41:29.015568 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:29.015542 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" containerID="cri-o://f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325" gracePeriod=30 Oct 30 01:41:33.657741 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.657716 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:41:33.693145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.693111 5055 generic.go:356] "Generic (PLEG): container finished" podID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerID="bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8" exitCode=0 Oct 30 01:41:33.693290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.693163 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" Oct 30 01:41:33.693290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.693192 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" event={"ID":"041906cb-d475-4df3-9a26-b3b422fe5eae","Type":"ContainerDied","Data":"bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8"} Oct 30 01:41:33.693290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.693262 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6" event={"ID":"041906cb-d475-4df3-9a26-b3b422fe5eae","Type":"ContainerDied","Data":"15df5cb26748105e668d7ae43719b9ce3afe0660dc1ca9759d4b785b3f469215"} Oct 30 01:41:33.693483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.693304 5055 scope.go:122] "RemoveContainer" containerID="bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8" Oct 30 01:41:33.707431 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.707414 5055 scope.go:122] "RemoveContainer" containerID="2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567" Oct 30 01:41:33.718105 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.718085 5055 scope.go:122] "RemoveContainer" containerID="bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8" Oct 30 01:41:33.718787 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:33.718405 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8\": container with ID starting with bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8 not found: ID does not exist" containerID="bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8" Oct 30 01:41:33.718787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.718438 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8"} err="failed to get container status \"bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8\": rpc error: code = NotFound desc = could not find container \"bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8\": container with ID starting with bfdc3539f115a278fc6e6f42a8851382ddc3adacffb940b975440ee103cf7ee8 not found: ID does not exist" Oct 30 01:41:33.718787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.718465 5055 scope.go:122] "RemoveContainer" containerID="2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567" Oct 30 01:41:33.718787 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:33.718710 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567\": container with ID starting with 2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567 not found: ID does not exist" containerID="2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567" Oct 30 01:41:33.718787 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.718742 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567"} err="failed to get container status \"2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567\": rpc error: code = NotFound desc = could not find container \"2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567\": container with ID starting with 2eac72afdc20aab61ab5f8ba847c0bd182e9201e5c7bb321fc55dcd867d4d567 not found: ID does not exist" Oct 30 01:41:33.725245 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725223 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-data\") pod \"041906cb-d475-4df3-9a26-b3b422fe5eae\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " Oct 30 01:41:33.725376 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725263 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-token\" (UniqueName: \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-token\") pod \"041906cb-d475-4df3-9a26-b3b422fe5eae\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " Oct 30 01:41:33.725376 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725341 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-socket\") pod \"041906cb-d475-4df3-9a26-b3b422fe5eae\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " Oct 30 01:41:33.725516 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725500 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/041906cb-d475-4df3-9a26-b3b422fe5eae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041906cb-d475-4df3-9a26-b3b422fe5eae-istiod-ca-cert\") pod \"041906cb-d475-4df3-9a26-b3b422fe5eae\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " Oct 30 01:41:33.725570 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725544 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-data" pod "041906cb-d475-4df3-9a26-b3b422fe5eae" (UID: "041906cb-d475-4df3-9a26-b3b422fe5eae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:33.725606 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725590 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-envoy\") pod \"041906cb-d475-4df3-9a26-b3b422fe5eae\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " Oct 30 01:41:33.725643 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725634 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-credential-socket\") pod \"041906cb-d475-4df3-9a26-b3b422fe5eae\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " Oct 30 01:41:33.725692 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725674 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-podinfo\") pod \"041906cb-d475-4df3-9a26-b3b422fe5eae\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " Oct 30 01:41:33.725760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725727 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-kube-api-access-j982p\" (UniqueName: \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-kube-api-access-j982p\") pod \"041906cb-d475-4df3-9a26-b3b422fe5eae\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " Oct 30 01:41:33.725819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725766 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-certs\") pod \"041906cb-d475-4df3-9a26-b3b422fe5eae\" (UID: \"041906cb-d475-4df3-9a26-b3b422fe5eae\") " Oct 30 01:41:33.726186 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725943 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/041906cb-d475-4df3-9a26-b3b422fe5eae-istiod-ca-cert" pod "041906cb-d475-4df3-9a26-b3b422fe5eae" (UID: "041906cb-d475-4df3-9a26-b3b422fe5eae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:33.726186 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.725998 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-socket" pod "041906cb-d475-4df3-9a26-b3b422fe5eae" (UID: "041906cb-d475-4df3-9a26-b3b422fe5eae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:33.726186 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.726156 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-credential-socket" pod "041906cb-d475-4df3-9a26-b3b422fe5eae" (UID: "041906cb-d475-4df3-9a26-b3b422fe5eae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:33.726395 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.726216 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-certs" pod "041906cb-d475-4df3-9a26-b3b422fe5eae" (UID: "041906cb-d475-4df3-9a26-b3b422fe5eae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:33.726450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.726430 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.726505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.726450 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.726505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.726464 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/041906cb-d475-4df3-9a26-b3b422fe5eae-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.726505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.726478 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.726505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.726491 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.728643 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.728610 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-kube-api-access-j982p" pod "041906cb-d475-4df3-9a26-b3b422fe5eae" (UID: "041906cb-d475-4df3-9a26-b3b422fe5eae"). InnerVolumeSpecName "kube-api-access-j982p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:33.728737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.728642 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-token" pod "041906cb-d475-4df3-9a26-b3b422fe5eae" (UID: "041906cb-d475-4df3-9a26-b3b422fe5eae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:33.728838 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.728811 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-podinfo" pod "041906cb-d475-4df3-9a26-b3b422fe5eae" (UID: "041906cb-d475-4df3-9a26-b3b422fe5eae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:41:33.731508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.731484 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-envoy" pod "041906cb-d475-4df3-9a26-b3b422fe5eae" (UID: "041906cb-d475-4df3-9a26-b3b422fe5eae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:33.828104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.828059 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.828104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.828097 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.828104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.828108 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j982p\" (UniqueName: \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-kube-api-access-j982p\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:33.828104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:33.828118 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/041906cb-d475-4df3-9a26-b3b422fe5eae-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:34.046448 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.046425 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6"] Oct 30 01:41:34.059284 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.059257 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-fc51e9-openshift-default-698b8b98c9-jskl6"] Oct 30 01:41:34.283809 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.283779 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:41:34.333617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.333590 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-envoy\") pod \"7f7826b4-dd81-4d91-8872-4079c418b79e\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " Oct 30 01:41:34.333997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.333678 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7826b4-dd81-4d91-8872-4079c418b79e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f7826b4-dd81-4d91-8872-4079c418b79e-istiod-ca-cert\") pod \"7f7826b4-dd81-4d91-8872-4079c418b79e\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " Oct 30 01:41:34.333997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.333712 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-socket\") pod \"7f7826b4-dd81-4d91-8872-4079c418b79e\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " Oct 30 01:41:34.333997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.333747 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-data\") pod \"7f7826b4-dd81-4d91-8872-4079c418b79e\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " Oct 30 01:41:34.333997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.333787 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-credential-socket\") pod \"7f7826b4-dd81-4d91-8872-4079c418b79e\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " Oct 30 01:41:34.333997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.333842 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-podinfo\") pod \"7f7826b4-dd81-4d91-8872-4079c418b79e\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " Oct 30 01:41:34.333997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.333881 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-kube-api-access-gzbwx\" (UniqueName: \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-kube-api-access-gzbwx\") pod \"7f7826b4-dd81-4d91-8872-4079c418b79e\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " Oct 30 01:41:34.333997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.333941 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-certs\") pod \"7f7826b4-dd81-4d91-8872-4079c418b79e\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " Oct 30 01:41:34.334417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.334132 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-data" pod "7f7826b4-dd81-4d91-8872-4079c418b79e" (UID: "7f7826b4-dd81-4d91-8872-4079c418b79e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:34.334417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.334222 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-token\" (UniqueName: \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-token\") pod \"7f7826b4-dd81-4d91-8872-4079c418b79e\" (UID: \"7f7826b4-dd81-4d91-8872-4079c418b79e\") " Oct 30 01:41:34.334883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.334858 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:34.335132 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.335107 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-certs" pod "7f7826b4-dd81-4d91-8872-4079c418b79e" (UID: "7f7826b4-dd81-4d91-8872-4079c418b79e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:34.335252 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.335171 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7826b4-dd81-4d91-8872-4079c418b79e-istiod-ca-cert" pod "7f7826b4-dd81-4d91-8872-4079c418b79e" (UID: "7f7826b4-dd81-4d91-8872-4079c418b79e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:34.335623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.335587 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-socket" pod "7f7826b4-dd81-4d91-8872-4079c418b79e" (UID: "7f7826b4-dd81-4d91-8872-4079c418b79e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:34.335776 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.335755 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-credential-socket" pod "7f7826b4-dd81-4d91-8872-4079c418b79e" (UID: "7f7826b4-dd81-4d91-8872-4079c418b79e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:34.336306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.336284 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-envoy" pod "7f7826b4-dd81-4d91-8872-4079c418b79e" (UID: "7f7826b4-dd81-4d91-8872-4079c418b79e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:34.337033 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.337005 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-podinfo" pod "7f7826b4-dd81-4d91-8872-4079c418b79e" (UID: "7f7826b4-dd81-4d91-8872-4079c418b79e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:41:34.337760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.337733 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-token" pod "7f7826b4-dd81-4d91-8872-4079c418b79e" (UID: "7f7826b4-dd81-4d91-8872-4079c418b79e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:34.338187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.338163 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-kube-api-access-gzbwx" pod "7f7826b4-dd81-4d91-8872-4079c418b79e" (UID: "7f7826b4-dd81-4d91-8872-4079c418b79e"). InnerVolumeSpecName "kube-api-access-gzbwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:34.436101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.436074 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:34.436101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.436101 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f7826b4-dd81-4d91-8872-4079c418b79e-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:34.436257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.436110 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:34.436257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.436119 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:34.436257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.436128 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:34.436257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.436136 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbwx\" (UniqueName: \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-kube-api-access-gzbwx\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:34.436257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.436145 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f7826b4-dd81-4d91-8872-4079c418b79e-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:34.436257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.436153 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f7826b4-dd81-4d91-8872-4079c418b79e-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:34.709502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.709468 5055 generic.go:356] "Generic (PLEG): container finished" podID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerID="f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325" exitCode=0 Oct 30 01:41:34.710380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.710352 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" event={"ID":"7f7826b4-dd81-4d91-8872-4079c418b79e","Type":"ContainerDied","Data":"f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325"} Oct 30 01:41:34.710523 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.710470 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" event={"ID":"7f7826b4-dd81-4d91-8872-4079c418b79e","Type":"ContainerDied","Data":"b28b86262c68c4ca2049c218342ba352db9ad04d3ae2bcd15088abbecfdfbe15"} Oct 30 01:41:34.710580 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.710525 5055 scope.go:122] "RemoveContainer" containerID="f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325" Oct 30 01:41:34.717161 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.710321 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255" Oct 30 01:41:34.723851 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.723828 5055 scope.go:122] "RemoveContainer" containerID="4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07" Oct 30 01:41:34.733087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.733068 5055 scope.go:122] "RemoveContainer" containerID="f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325" Oct 30 01:41:34.733429 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:34.733402 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325\": container with ID starting with f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325 not found: ID does not exist" containerID="f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325" Oct 30 01:41:34.733502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.733440 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325"} err="failed to get container status \"f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325\": rpc error: code = NotFound desc = could not find container \"f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325\": container with ID starting with f4193980809e60bd7f6b9e3065272b061bc1d25d37c3c365b12f104a943c1325 not found: ID does not exist" Oct 30 01:41:34.733502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.733462 5055 scope.go:122] "RemoveContainer" containerID="4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07" Oct 30 01:41:34.733749 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:34.733730 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07\": container with ID starting with 4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07 not found: ID does not exist" containerID="4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07" Oct 30 01:41:34.733820 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.733758 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07"} err="failed to get container status \"4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07\": rpc error: code = NotFound desc = could not find container \"4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07\": container with ID starting with 4033f05932615d2312d6daef4ae84dc96678f1be0f62429ba05d132397c8be07 not found: ID does not exist" Oct 30 01:41:34.779745 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.779720 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255"] Oct 30 01:41:34.792023 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.792003 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d9ede0-openshift-default-5bc96ff9cc-s6255"] Oct 30 01:41:34.820212 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.820190 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" path="/var/lib/kubelet/pods/041906cb-d475-4df3-9a26-b3b422fe5eae/volumes" Oct 30 01:41:34.820714 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:34.820698 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" path="/var/lib/kubelet/pods/7f7826b4-dd81-4d91-8872-4079c418b79e/volumes" Oct 30 01:41:37.929260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:37.929223 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd"] Oct 30 01:41:37.930776 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:37.930743 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" podUID="84e365f7-3976-4260-9a0f-cb57aa05cbbe" containerName="payload-processing" containerID="cri-o://ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a" gracePeriod=30 Oct 30 01:41:37.945924 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:37.945900 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb"] Oct 30 01:41:37.946102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:37.946083 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" podUID="d4250e3b-3363-48ff-8e1b-f86d82f411cc" containerName="payload-pre-processing" containerID="cri-o://094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd" gracePeriod=30 Oct 30 01:41:38.434366 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.430666 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:41:38.464017 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.463993 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:41:38.515918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.515786 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e365f7-3976-4260-9a0f-cb57aa05cbbe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e365f7-3976-4260-9a0f-cb57aa05cbbe-plugins-config-volume\") pod \"84e365f7-3976-4260-9a0f-cb57aa05cbbe\" (UID: \"84e365f7-3976-4260-9a0f-cb57aa05cbbe\") " Oct 30 01:41:38.515918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.515883 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e365f7-3976-4260-9a0f-cb57aa05cbbe-kube-api-access-mw7km\" (UniqueName: \"kubernetes.io/projected/84e365f7-3976-4260-9a0f-cb57aa05cbbe-kube-api-access-mw7km\") pod \"84e365f7-3976-4260-9a0f-cb57aa05cbbe\" (UID: \"84e365f7-3976-4260-9a0f-cb57aa05cbbe\") " Oct 30 01:41:38.516195 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.515976 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4250e3b-3363-48ff-8e1b-f86d82f411cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4250e3b-3363-48ff-8e1b-f86d82f411cc-plugins-config-volume\") pod \"d4250e3b-3363-48ff-8e1b-f86d82f411cc\" (UID: \"d4250e3b-3363-48ff-8e1b-f86d82f411cc\") " Oct 30 01:41:38.518371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.516366 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4250e3b-3363-48ff-8e1b-f86d82f411cc-kube-api-access-z67qj\" (UniqueName: \"kubernetes.io/projected/d4250e3b-3363-48ff-8e1b-f86d82f411cc-kube-api-access-z67qj\") pod \"d4250e3b-3363-48ff-8e1b-f86d82f411cc\" (UID: \"d4250e3b-3363-48ff-8e1b-f86d82f411cc\") " Oct 30 01:41:38.519683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.519652 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e365f7-3976-4260-9a0f-cb57aa05cbbe-plugins-config-volume" pod "84e365f7-3976-4260-9a0f-cb57aa05cbbe" (UID: "84e365f7-3976-4260-9a0f-cb57aa05cbbe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:38.519813 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.519775 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4250e3b-3363-48ff-8e1b-f86d82f411cc-plugins-config-volume" pod "d4250e3b-3363-48ff-8e1b-f86d82f411cc" (UID: "d4250e3b-3363-48ff-8e1b-f86d82f411cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:38.523619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.523590 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4250e3b-3363-48ff-8e1b-f86d82f411cc-kube-api-access-z67qj" pod "d4250e3b-3363-48ff-8e1b-f86d82f411cc" (UID: "d4250e3b-3363-48ff-8e1b-f86d82f411cc"). InnerVolumeSpecName "kube-api-access-z67qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:38.524159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.524137 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z67qj\" (UniqueName: \"kubernetes.io/projected/d4250e3b-3363-48ff-8e1b-f86d82f411cc-kube-api-access-z67qj\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:38.524216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.524169 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84e365f7-3976-4260-9a0f-cb57aa05cbbe-plugins-config-volume\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:38.524216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.524183 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4250e3b-3363-48ff-8e1b-f86d82f411cc-plugins-config-volume\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:38.530420 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.530394 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e365f7-3976-4260-9a0f-cb57aa05cbbe-kube-api-access-mw7km" pod "84e365f7-3976-4260-9a0f-cb57aa05cbbe" (UID: "84e365f7-3976-4260-9a0f-cb57aa05cbbe"). InnerVolumeSpecName "kube-api-access-mw7km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:38.625785 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.625758 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw7km\" (UniqueName: \"kubernetes.io/projected/84e365f7-3976-4260-9a0f-cb57aa05cbbe-kube-api-access-mw7km\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:38.740794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.740747 5055 generic.go:356] "Generic (PLEG): container finished" podID="d4250e3b-3363-48ff-8e1b-f86d82f411cc" containerID="094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd" exitCode=0 Oct 30 01:41:38.740942 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.740872 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" Oct 30 01:41:38.740942 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.740888 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" event={"ID":"d4250e3b-3363-48ff-8e1b-f86d82f411cc","Type":"ContainerDied","Data":"094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd"} Oct 30 01:41:38.741060 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.740985 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb" event={"ID":"d4250e3b-3363-48ff-8e1b-f86d82f411cc","Type":"ContainerDied","Data":"4effeb729092954364e48c74c52607e1bbb1aaedde54466712509aeb162e7475"} Oct 30 01:41:38.741060 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.741018 5055 scope.go:122] "RemoveContainer" containerID="094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd" Oct 30 01:41:38.742402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.742380 5055 generic.go:356] "Generic (PLEG): container finished" podID="84e365f7-3976-4260-9a0f-cb57aa05cbbe" containerID="ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a" exitCode=0 Oct 30 01:41:38.742521 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.742491 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" event={"ID":"84e365f7-3976-4260-9a0f-cb57aa05cbbe","Type":"ContainerDied","Data":"ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a"} Oct 30 01:41:38.742521 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.742513 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" Oct 30 01:41:38.742621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.742531 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd" event={"ID":"84e365f7-3976-4260-9a0f-cb57aa05cbbe","Type":"ContainerDied","Data":"ba21df2f81a0eeebcc414c5a7c24a05a9e6dbeb4bc31dc68cdf068c0c8e99f24"} Oct 30 01:41:38.752991 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.752972 5055 scope.go:122] "RemoveContainer" containerID="094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd" Oct 30 01:41:38.753258 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:38.753234 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd\": container with ID starting with 094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd not found: ID does not exist" containerID="094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd" Oct 30 01:41:38.753383 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.753265 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd"} err="failed to get container status \"094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd\": rpc error: code = NotFound desc = could not find container \"094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd\": container with ID starting with 094e6d999187cd54c9b84f3e2e6bd07e012b568c36936ea849b6a9f724ee68dd not found: ID does not exist" Oct 30 01:41:38.753383 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.753282 5055 scope.go:122] "RemoveContainer" containerID="ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a" Oct 30 01:41:38.764632 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.764610 5055 scope.go:122] "RemoveContainer" containerID="ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a" Oct 30 01:41:38.764974 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:38.764957 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a\": container with ID starting with ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a not found: ID does not exist" containerID="ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a" Oct 30 01:41:38.765140 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.764985 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a"} err="failed to get container status \"ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a\": rpc error: code = NotFound desc = could not find container \"ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a\": container with ID starting with ec0713231e19f734164809f629a16597d3dbea01be457607f881da59c519849a not found: ID does not exist" Oct 30 01:41:38.797766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.797686 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd"] Oct 30 01:41:38.803513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.803485 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f7bdcc-5b94c49996-bfjkd"] Oct 30 01:41:38.821899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.821865 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e365f7-3976-4260-9a0f-cb57aa05cbbe" path="/var/lib/kubelet/pods/84e365f7-3976-4260-9a0f-cb57aa05cbbe/volumes" Oct 30 01:41:38.836973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.836949 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb"] Oct 30 01:41:38.841062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:38.841039 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-5dbdc89559-2bxdb"] Oct 30 01:41:40.822155 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:40.822108 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4250e3b-3363-48ff-8e1b-f86d82f411cc" path="/var/lib/kubelet/pods/d4250e3b-3363-48ff-8e1b-f86d82f411cc/volumes" Oct 30 01:41:44.920235 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:44.920203 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6"] Oct 30 01:41:44.920662 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:44.920409 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" podUID="25014618-d292-485b-85d7-80a086f2eb0f" containerName="maas-api" containerID="cri-o://e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad" gracePeriod=30 Oct 30 01:41:45.298492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.298460 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:41:45.420347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.417030 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd"] Oct 30 01:41:45.420347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.417268 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" containerID="cri-o://1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f" gracePeriod=30 Oct 30 01:41:45.428351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.426882 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-metrics-certs\") pod \"25014618-d292-485b-85d7-80a086f2eb0f\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " Oct 30 01:41:45.428351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.427445 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25014618-d292-485b-85d7-80a086f2eb0f-kube-api-access-jtbzf\" (UniqueName: \"kubernetes.io/projected/25014618-d292-485b-85d7-80a086f2eb0f-kube-api-access-jtbzf\") pod \"25014618-d292-485b-85d7-80a086f2eb0f\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " Oct 30 01:41:45.428351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.427667 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25014618-d292-485b-85d7-80a086f2eb0f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25014618-d292-485b-85d7-80a086f2eb0f-openshift-service-ca\") pod \"25014618-d292-485b-85d7-80a086f2eb0f\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " Oct 30 01:41:45.428351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.427730 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-maas-api-tls\") pod \"25014618-d292-485b-85d7-80a086f2eb0f\" (UID: \"25014618-d292-485b-85d7-80a086f2eb0f\") " Oct 30 01:41:45.430014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.428953 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25014618-d292-485b-85d7-80a086f2eb0f-openshift-service-ca" pod "25014618-d292-485b-85d7-80a086f2eb0f" (UID: "25014618-d292-485b-85d7-80a086f2eb0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:45.434348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.430443 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-metrics-certs" pod "25014618-d292-485b-85d7-80a086f2eb0f" (UID: "25014618-d292-485b-85d7-80a086f2eb0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:45.434348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.431504 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25014618-d292-485b-85d7-80a086f2eb0f-kube-api-access-jtbzf" pod "25014618-d292-485b-85d7-80a086f2eb0f" (UID: "25014618-d292-485b-85d7-80a086f2eb0f"). InnerVolumeSpecName "kube-api-access-jtbzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:45.441373 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.435610 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-maas-api-tls" pod "25014618-d292-485b-85d7-80a086f2eb0f" (UID: "25014618-d292-485b-85d7-80a086f2eb0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:45.539406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.529548 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:45.539406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.529592 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtbzf\" (UniqueName: \"kubernetes.io/projected/25014618-d292-485b-85d7-80a086f2eb0f-kube-api-access-jtbzf\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:45.539406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.529610 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25014618-d292-485b-85d7-80a086f2eb0f-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:45.539406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.529623 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25014618-d292-485b-85d7-80a086f2eb0f-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:45.796375 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.796243 5055 generic.go:356] "Generic (PLEG): container finished" podID="25014618-d292-485b-85d7-80a086f2eb0f" containerID="e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad" exitCode=0 Oct 30 01:41:45.796536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.796365 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" event={"ID":"25014618-d292-485b-85d7-80a086f2eb0f","Type":"ContainerDied","Data":"e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad"} Oct 30 01:41:45.796536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.796423 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" event={"ID":"25014618-d292-485b-85d7-80a086f2eb0f","Type":"ContainerDied","Data":"983a3fe99a884b6a3497a00cf7c92929359ebec55424858ef60ad97ca03f55c6"} Oct 30 01:41:45.796536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.796370 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6" Oct 30 01:41:45.796536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.796450 5055 scope.go:122] "RemoveContainer" containerID="e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad" Oct 30 01:41:45.809071 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.809050 5055 scope.go:122] "RemoveContainer" containerID="e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad" Oct 30 01:41:45.809393 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:45.809365 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad\": container with ID starting with e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad not found: ID does not exist" containerID="e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad" Oct 30 01:41:45.809508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.809404 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad"} err="failed to get container status \"e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad\": rpc error: code = NotFound desc = could not find container \"e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad\": container with ID starting with e4cdd28675d35e8b27919a9b681fc35af235b65acf7fbe8f30317f3e3d3b09ad not found: ID does not exist" Oct 30 01:41:45.850415 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.850293 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6"] Oct 30 01:41:45.853901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:45.853874 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d8eb06-655d6f9d6b-65ln6"] Oct 30 01:41:46.819872 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:46.819845 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25014618-d292-485b-85d7-80a086f2eb0f" path="/var/lib/kubelet/pods/25014618-d292-485b-85d7-80a086f2eb0f/volumes" Oct 30 01:41:49.224350 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:49.224287 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:41:50.703371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.703308 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:41:50.714417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.714388 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-certs\") pod \"81a299b6-5136-4faa-99e6-4a9d0242869d\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " Oct 30 01:41:50.714536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.714490 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-data\") pod \"81a299b6-5136-4faa-99e6-4a9d0242869d\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " Oct 30 01:41:50.714577 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.714551 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-token\" (UniqueName: \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-token\") pod \"81a299b6-5136-4faa-99e6-4a9d0242869d\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " Oct 30 01:41:50.714634 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.714618 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-podinfo\") pod \"81a299b6-5136-4faa-99e6-4a9d0242869d\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " Oct 30 01:41:50.714700 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.714686 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-socket\") pod \"81a299b6-5136-4faa-99e6-4a9d0242869d\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " Oct 30 01:41:50.714749 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.714616 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-certs" pod "81a299b6-5136-4faa-99e6-4a9d0242869d" (UID: "81a299b6-5136-4faa-99e6-4a9d0242869d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:50.714749 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.714718 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-kube-api-access-s9hp6\" (UniqueName: \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-kube-api-access-s9hp6\") pod \"81a299b6-5136-4faa-99e6-4a9d0242869d\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " Oct 30 01:41:50.714854 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.714777 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-credential-socket\") pod \"81a299b6-5136-4faa-99e6-4a9d0242869d\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " Oct 30 01:41:50.714854 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.714807 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a299b6-5136-4faa-99e6-4a9d0242869d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a299b6-5136-4faa-99e6-4a9d0242869d-istiod-ca-cert\") pod \"81a299b6-5136-4faa-99e6-4a9d0242869d\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " Oct 30 01:41:50.714961 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.714931 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-envoy\") pod \"81a299b6-5136-4faa-99e6-4a9d0242869d\" (UID: \"81a299b6-5136-4faa-99e6-4a9d0242869d\") " Oct 30 01:41:50.715031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.715006 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-socket" pod "81a299b6-5136-4faa-99e6-4a9d0242869d" (UID: "81a299b6-5136-4faa-99e6-4a9d0242869d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:50.715182 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.715157 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-credential-socket" pod "81a299b6-5136-4faa-99e6-4a9d0242869d" (UID: "81a299b6-5136-4faa-99e6-4a9d0242869d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:50.715730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.715702 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:50.715836 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.715733 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:50.715836 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.715748 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:50.715979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.715910 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a299b6-5136-4faa-99e6-4a9d0242869d-istiod-ca-cert" pod "81a299b6-5136-4faa-99e6-4a9d0242869d" (UID: "81a299b6-5136-4faa-99e6-4a9d0242869d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:50.715979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.715964 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-data" pod "81a299b6-5136-4faa-99e6-4a9d0242869d" (UID: "81a299b6-5136-4faa-99e6-4a9d0242869d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:50.717097 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.717068 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-token" pod "81a299b6-5136-4faa-99e6-4a9d0242869d" (UID: "81a299b6-5136-4faa-99e6-4a9d0242869d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:50.717478 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.717451 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-podinfo" pod "81a299b6-5136-4faa-99e6-4a9d0242869d" (UID: "81a299b6-5136-4faa-99e6-4a9d0242869d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:41:50.717598 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.717514 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-kube-api-access-s9hp6" pod "81a299b6-5136-4faa-99e6-4a9d0242869d" (UID: "81a299b6-5136-4faa-99e6-4a9d0242869d"). InnerVolumeSpecName "kube-api-access-s9hp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:50.717663 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.717602 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-envoy" pod "81a299b6-5136-4faa-99e6-4a9d0242869d" (UID: "81a299b6-5136-4faa-99e6-4a9d0242869d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:50.817669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.817642 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:50.817897 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.817872 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:50.817984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.817901 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:50.817984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.817916 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9hp6\" (UniqueName: \"kubernetes.io/projected/81a299b6-5136-4faa-99e6-4a9d0242869d-kube-api-access-s9hp6\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:50.817984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.817931 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81a299b6-5136-4faa-99e6-4a9d0242869d-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:50.817984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.817944 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81a299b6-5136-4faa-99e6-4a9d0242869d-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:50.835042 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.835021 5055 generic.go:356] "Generic (PLEG): container finished" podID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerID="1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f" exitCode=0 Oct 30 01:41:50.835148 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.835062 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" Oct 30 01:41:50.835148 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.835103 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" event={"ID":"81a299b6-5136-4faa-99e6-4a9d0242869d","Type":"ContainerDied","Data":"1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f"} Oct 30 01:41:50.835238 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.835147 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd" event={"ID":"81a299b6-5136-4faa-99e6-4a9d0242869d","Type":"ContainerDied","Data":"d29b5d7baa75520422db6285c74185d9f41dee2d46d30f2de728411d1a74ff07"} Oct 30 01:41:50.835238 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.835175 5055 scope.go:122] "RemoveContainer" containerID="1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f" Oct 30 01:41:50.845113 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.845091 5055 scope.go:122] "RemoveContainer" containerID="303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1" Oct 30 01:41:50.855445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.855416 5055 scope.go:122] "RemoveContainer" containerID="1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f" Oct 30 01:41:50.855793 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:50.855771 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f\": container with ID starting with 1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f not found: ID does not exist" containerID="1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f" Oct 30 01:41:50.855899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.855805 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f"} err="failed to get container status \"1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f\": rpc error: code = NotFound desc = could not find container \"1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f\": container with ID starting with 1b9480f957b376caffdafeff481ac0b1c48bff6e57a7f39009c1ab0e044c336f not found: ID does not exist" Oct 30 01:41:50.855899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.855824 5055 scope.go:122] "RemoveContainer" containerID="303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1" Oct 30 01:41:50.856164 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:41:50.856135 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1\": container with ID starting with 303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1 not found: ID does not exist" containerID="303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1" Oct 30 01:41:50.856235 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.856164 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1"} err="failed to get container status \"303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1\": rpc error: code = NotFound desc = could not find container \"303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1\": container with ID starting with 303012f5689544da356f3d341a4dcf4141d4be18a14f246a7fb28fe180dbe9a1 not found: ID does not exist" Oct 30 01:41:50.890347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.888754 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd"] Oct 30 01:41:50.894277 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:50.894249 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-f7bdcc-openshift-default-5d5cc854df-l5xsd"] Oct 30 01:41:52.590503 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:52.590263 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn"] Oct 30 01:41:52.591011 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:52.590589 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" containerID="cri-o://1a0b18a2b67c3e4b9860fd323e0720965d8c20620d866987371a3e6ecb1672e3" gracePeriod=30 Oct 30 01:41:52.822624 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:52.822592 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" path="/var/lib/kubelet/pods/81a299b6-5136-4faa-99e6-4a9d0242869d/volumes" Oct 30 01:41:53.333526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:53.333491 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:41:53.333702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:53.333567 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:41:54.100351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.099392 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd"] Oct 30 01:41:54.100351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100272 5055 cpu_manager.go:414] "containerMap: removing container" podUID="1736cd25-3aa2-4ea0-8557-8f58096599df" containerName="maas-api" Oct 30 01:41:54.100351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100292 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1736cd25-3aa2-4ea0-8557-8f58096599df" containerName="maas-api" Oct 30 01:41:54.100351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100307 5055 cpu_manager.go:414] "containerMap: removing container" podUID="25014618-d292-485b-85d7-80a086f2eb0f" containerName="maas-api" Oct 30 01:41:54.100351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100316 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25014618-d292-485b-85d7-80a086f2eb0f" containerName="maas-api" Oct 30 01:41:54.100351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100353 5055 cpu_manager.go:414] "containerMap: removing container" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" Oct 30 01:41:54.100351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100361 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100382 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d4250e3b-3363-48ff-8e1b-f86d82f411cc" containerName="payload-pre-processing" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100389 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4250e3b-3363-48ff-8e1b-f86d82f411cc" containerName="payload-pre-processing" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100408 5055 cpu_manager.go:414] "containerMap: removing container" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100415 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100442 5055 cpu_manager.go:414] "containerMap: removing container" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100449 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100459 5055 cpu_manager.go:414] "containerMap: removing container" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100466 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100481 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100489 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100498 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100505 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100526 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100532 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100547 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100554 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100572 5055 cpu_manager.go:414] "containerMap: removing container" podUID="db4553d4-7bf8-4fdc-b6a9-2036c1300826" containerName="maas-api" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100580 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4553d4-7bf8-4fdc-b6a9-2036c1300826" containerName="maas-api" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100597 5055 cpu_manager.go:414] "containerMap: removing container" podUID="84e365f7-3976-4260-9a0f-cb57aa05cbbe" containerName="payload-processing" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100604 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e365f7-3976-4260-9a0f-cb57aa05cbbe" containerName="payload-processing" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100752 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100766 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a299b6-5136-4faa-99e6-4a9d0242869d" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100776 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100786 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100796 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f0697a-4001-46eb-8bf6-83200b8d0062" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100805 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7826b4-dd81-4d91-8872-4079c418b79e" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100814 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4553d4-7bf8-4fdc-b6a9-2036c1300826" containerName="maas-api" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100825 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e365f7-3976-4260-9a0f-cb57aa05cbbe" containerName="payload-processing" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100837 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="1736cd25-3aa2-4ea0-8557-8f58096599df" containerName="maas-api" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100845 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="25014618-d292-485b-85d7-80a086f2eb0f" containerName="maas-api" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100855 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100864 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4250e3b-3363-48ff-8e1b-f86d82f411cc" containerName="payload-pre-processing" Oct 30 01:41:54.101101 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.100874 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="041906cb-d475-4df3-9a26-b3b422fe5eae" containerName="istio-proxy" Oct 30 01:41:54.129362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.106552 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.129362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.122295 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ea0fc4-openshift-default-dockercfg-l4jxq\"" Oct 30 01:41:54.143067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.143044 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd"] Oct 30 01:41:54.166072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.165090 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-socket\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.166072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.165150 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46nns\" (UniqueName: \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-kube-api-access-46nns\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.166072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.165183 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-certs\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.166072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.165223 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-podinfo\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.166072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.165314 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-data\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.166072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.165365 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-token\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.166072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.165390 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istiod-ca-cert\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.166072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.165454 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-envoy\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.166072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.165521 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-credential-socket\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.288350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.288215 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-data\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.288350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.288268 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-token\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.288350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.288296 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istiod-ca-cert\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.288712 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.288444 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-envoy\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.288712 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.288561 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-credential-socket\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.288712 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.288633 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-socket\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.288712 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.288679 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46nns\" (UniqueName: \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-kube-api-access-46nns\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.288712 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.288705 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-certs\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.288984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.288753 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-podinfo\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.292347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.289113 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-credential-socket\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.292347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.289763 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istiod-ca-cert\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.301553 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.292851 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-socket\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.301553 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.293102 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-certs\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.306845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.306789 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-envoy\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.307786 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.307762 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-data\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.308908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.308858 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-podinfo\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.316357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.312787 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-token\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.331352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.330256 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46nns\" (UniqueName: \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-kube-api-access-46nns\") pod \"e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.427192 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.427099 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:54.554298 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.554265 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7"] Oct 30 01:41:54.605391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.604485 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7"] Oct 30 01:41:54.605391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.604604 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.621000 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.620466 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-c22e6b-openshift-default-dockercfg-6zd7b\"" Oct 30 01:41:54.718361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.717710 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-podinfo\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.718361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.717857 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-data\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.718361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.717993 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-credential-socket\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.718361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.718066 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-envoy\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.718361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.718125 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgxdp\" (UniqueName: \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-kube-api-access-lgxdp\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.718361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.718198 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-token\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.718361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.718276 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-certs\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.718883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.718506 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istiod-ca-cert\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.718883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.718609 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-socket\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.823281 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istiod-ca-cert\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.823380 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-socket\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.823466 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-podinfo\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.823512 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-data\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.823563 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-credential-socket\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.823592 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-envoy\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.823631 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgxdp\" (UniqueName: \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-kube-api-access-lgxdp\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.823671 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-token\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.823729 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-certs\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.824133 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-certs\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.825630 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.825409 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istiod-ca-cert\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.826045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.825907 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-socket\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.829394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.829255 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-podinfo\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.830218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.829730 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-data\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.830218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.829972 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-credential-socket\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.835719 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.832030 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-envoy\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.838594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.837718 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:41:54.838594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.837795 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:41:54.838594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.837834 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:41:54.843802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.843784 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd"] Oct 30 01:41:54.856393 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.851251 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-token\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.856393 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.851514 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgxdp\" (UniqueName: \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-kube-api-access-lgxdp\") pod \"e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:54.865102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.864301 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" event={"ID":"c94cdcb7-32d4-437f-af1f-df2a30c79d3b","Type":"ContainerStarted","Data":"13bdfcd3304d8892ce839ce3ada8a5922b807babc037f7d8e6e8e317ed0f1a7e"} Oct 30 01:41:54.928350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:54.928094 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:55.268845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:55.268810 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7"] Oct 30 01:41:55.269412 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:41:55.269029 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bafa687_8080_4ee5_9388_fb0d6d74ffc1.slice/crio-2a3e2af55de785494886f7c12ede25ab7bf09f534fef7e669fcb6acb442c28bc WatchSource:0}: Error finding container 2a3e2af55de785494886f7c12ede25ab7bf09f534fef7e669fcb6acb442c28bc: Status 404 returned error can't find the container with id 2a3e2af55de785494886f7c12ede25ab7bf09f534fef7e669fcb6acb442c28bc Oct 30 01:41:55.272289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:55.272255 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:41:55.272405 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:55.272355 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:41:55.272405 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:55.272403 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:41:55.875111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:55.875076 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" event={"ID":"1bafa687-8080-4ee5-9388-fb0d6d74ffc1","Type":"ContainerStarted","Data":"9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad"} Oct 30 01:41:55.875377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:55.875353 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" event={"ID":"1bafa687-8080-4ee5-9388-fb0d6d74ffc1","Type":"ContainerStarted","Data":"2a3e2af55de785494886f7c12ede25ab7bf09f534fef7e669fcb6acb442c28bc"} Oct 30 01:41:55.877110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:55.877053 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" event={"ID":"c94cdcb7-32d4-437f-af1f-df2a30c79d3b","Type":"ContainerStarted","Data":"1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39"} Oct 30 01:41:55.920921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:55.920864 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" podStartSLOduration=1.920845052 podStartE2EDuration="1.920845052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:55.916283474 +0000 UTC m=+1483.237240514" watchObservedRunningTime="2026-10-30 01:41:55.920845052 +0000 UTC m=+1483.241802093" Oct 30 01:41:55.930906 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:55.930867 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:55.961632 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:55.961585 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" podStartSLOduration=1.9615693109999999 podStartE2EDuration="1.961569311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:55.959822787 +0000 UTC m=+1483.280779829" watchObservedRunningTime="2026-10-30 01:41:55.961569311 +0000 UTC m=+1483.282526352" Oct 30 01:41:56.427810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:56.427780 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:56.433770 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:56.433719 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:56.884283 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:56.884254 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:56.885211 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:56.885184 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:41:56.938593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:56.938566 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:57.392088 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.392052 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt"] Oct 30 01:41:57.397526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.397498 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.416408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.416382 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt"] Oct 30 01:41:57.416831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.416733 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-3fde1c-serving-cert\"" Oct 30 01:41:57.465811 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.465778 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bcvr\" (UniqueName: \"kubernetes.io/projected/857ef08b-a4e1-404b-9c40-4a0dc5744e13-kube-api-access-5bcvr\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.466530 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.466502 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-metrics-certs\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.466677 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.466628 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.466764 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.466688 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/857ef08b-a4e1-404b-9c40-4a0dc5744e13-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.567743 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.567702 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bcvr\" (UniqueName: \"kubernetes.io/projected/857ef08b-a4e1-404b-9c40-4a0dc5744e13-kube-api-access-5bcvr\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.567928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.567806 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-metrics-certs\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.567977 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.567932 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.568033 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.567990 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/857ef08b-a4e1-404b-9c40-4a0dc5744e13-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.568776 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.568748 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/857ef08b-a4e1-404b-9c40-4a0dc5744e13-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.570557 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.570533 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-maas-api-tls\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.574829 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.574805 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-metrics-certs\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.590253 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.590221 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bcvr\" (UniqueName: \"kubernetes.io/projected/857ef08b-a4e1-404b-9c40-4a0dc5744e13-kube-api-access-5bcvr\") pod \"maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.712953 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.712921 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:57.894370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.894317 5055 generic.go:356] "Generic (PLEG): container finished" podID="316b9154-8e84-4e72-9250-d67591d60ed5" containerID="1a0b18a2b67c3e4b9860fd323e0720965d8c20620d866987371a3e6ecb1672e3" exitCode=0 Oct 30 01:41:57.895257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.895231 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" event={"ID":"316b9154-8e84-4e72-9250-d67591d60ed5","Type":"ContainerDied","Data":"1a0b18a2b67c3e4b9860fd323e0720965d8c20620d866987371a3e6ecb1672e3"} Oct 30 01:41:57.895403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.895292 5055 scope.go:122] "RemoveContainer" containerID="12ccbc63fbc85733eb094094f349dd5c2619e2b18fc7aa336a8e60c5f4d93958" Oct 30 01:41:57.896437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.896404 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:57.897578 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.897283 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:41:57.908737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.908714 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:41:57.992437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.991962 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-envoy\") pod \"316b9154-8e84-4e72-9250-d67591d60ed5\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " Oct 30 01:41:57.992437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.992098 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-socket\") pod \"316b9154-8e84-4e72-9250-d67591d60ed5\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " Oct 30 01:41:57.992437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.992128 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-certs\") pod \"316b9154-8e84-4e72-9250-d67591d60ed5\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " Oct 30 01:41:57.992437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.992208 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/316b9154-8e84-4e72-9250-d67591d60ed5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/316b9154-8e84-4e72-9250-d67591d60ed5-istiod-ca-cert\") pod \"316b9154-8e84-4e72-9250-d67591d60ed5\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.992851 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/316b9154-8e84-4e72-9250-d67591d60ed5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/316b9154-8e84-4e72-9250-d67591d60ed5-istio-podinfo\") pod \"316b9154-8e84-4e72-9250-d67591d60ed5\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.992902 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-socket" pod "316b9154-8e84-4e72-9250-d67591d60ed5" (UID: "316b9154-8e84-4e72-9250-d67591d60ed5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.992957 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-credential-socket\") pod \"316b9154-8e84-4e72-9250-d67591d60ed5\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.992992 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-data\") pod \"316b9154-8e84-4e72-9250-d67591d60ed5\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.993051 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-kube-api-access-sk9vb\" (UniqueName: \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-kube-api-access-sk9vb\") pod \"316b9154-8e84-4e72-9250-d67591d60ed5\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.993080 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316b9154-8e84-4e72-9250-d67591d60ed5-istiod-ca-cert" pod "316b9154-8e84-4e72-9250-d67591d60ed5" (UID: "316b9154-8e84-4e72-9250-d67591d60ed5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.993093 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-istio-token\" (UniqueName: \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-istio-token\") pod \"316b9154-8e84-4e72-9250-d67591d60ed5\" (UID: \"316b9154-8e84-4e72-9250-d67591d60ed5\") " Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.993590 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-data" pod "316b9154-8e84-4e72-9250-d67591d60ed5" (UID: "316b9154-8e84-4e72-9250-d67591d60ed5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.993973 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.993993 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/316b9154-8e84-4e72-9250-d67591d60ed5-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.994005 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:57.994803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.994269 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-credential-socket" pod "316b9154-8e84-4e72-9250-d67591d60ed5" (UID: "316b9154-8e84-4e72-9250-d67591d60ed5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:57.995483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.994999 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-certs" pod "316b9154-8e84-4e72-9250-d67591d60ed5" (UID: "316b9154-8e84-4e72-9250-d67591d60ed5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:57.997847 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:57.997807 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-istio-token" pod "316b9154-8e84-4e72-9250-d67591d60ed5" (UID: "316b9154-8e84-4e72-9250-d67591d60ed5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:58.001801 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.001764 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/316b9154-8e84-4e72-9250-d67591d60ed5-istio-podinfo" pod "316b9154-8e84-4e72-9250-d67591d60ed5" (UID: "316b9154-8e84-4e72-9250-d67591d60ed5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:41:58.001945 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.001913 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-envoy" pod "316b9154-8e84-4e72-9250-d67591d60ed5" (UID: "316b9154-8e84-4e72-9250-d67591d60ed5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:58.001945 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.001935 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-kube-api-access-sk9vb" pod "316b9154-8e84-4e72-9250-d67591d60ed5" (UID: "316b9154-8e84-4e72-9250-d67591d60ed5"). InnerVolumeSpecName "kube-api-access-sk9vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:58.060690 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:41:58.060659 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod857ef08b_a4e1_404b_9c40_4a0dc5744e13.slice/crio-287450bca25f7d6a1fa00e26ecbc65bfff342aff1eb2a8bdbf54870a7656a3ec WatchSource:0}: Error finding container 287450bca25f7d6a1fa00e26ecbc65bfff342aff1eb2a8bdbf54870a7656a3ec: Status 404 returned error can't find the container with id 287450bca25f7d6a1fa00e26ecbc65bfff342aff1eb2a8bdbf54870a7656a3ec Oct 30 01:41:58.064120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.064095 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt"] Oct 30 01:41:58.095423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.095399 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:58.095423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.095423 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/316b9154-8e84-4e72-9250-d67591d60ed5-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:58.095594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.095433 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:58.095594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.095442 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk9vb\" (UniqueName: \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-kube-api-access-sk9vb\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:58.095594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.095453 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/316b9154-8e84-4e72-9250-d67591d60ed5-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:58.095594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.095461 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/316b9154-8e84-4e72-9250-d67591d60ed5-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:41:58.907478 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.907436 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" event={"ID":"857ef08b-a4e1-404b-9c40-4a0dc5744e13","Type":"ContainerStarted","Data":"7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c"} Oct 30 01:41:58.907822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.907492 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" event={"ID":"857ef08b-a4e1-404b-9c40-4a0dc5744e13","Type":"ContainerStarted","Data":"287450bca25f7d6a1fa00e26ecbc65bfff342aff1eb2a8bdbf54870a7656a3ec"} Oct 30 01:41:58.908400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.908383 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:41:58.910391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.910358 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" event={"ID":"316b9154-8e84-4e72-9250-d67591d60ed5","Type":"ContainerDied","Data":"ab0841155c51e750b088c86320c3f182a33ea1ee2110281d1289da6a0cd577cc"} Oct 30 01:41:58.910490 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.910420 5055 scope.go:122] "RemoveContainer" containerID="1a0b18a2b67c3e4b9860fd323e0720965d8c20620d866987371a3e6ecb1672e3" Oct 30 01:41:58.910532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.910518 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn" Oct 30 01:41:58.961940 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:58.961888 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" podStartSLOduration=1.528400332 podStartE2EDuration="1.961874647s" totalImagesPullingTime="433.474315ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:58.959381864 +0000 UTC m=+1486.280338904" watchObservedRunningTime="2026-10-30 01:41:58.961874647 +0000 UTC m=+1486.282831686" Oct 30 01:41:59.037039 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:59.037009 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn"] Oct 30 01:41:59.046644 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:41:59.046615 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d8eb06-openshift-default-c9b679887-68hxn"] Oct 30 01:42:00.651371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.651340 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts"] Oct 30 01:42:00.651954 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.651940 5055 cpu_manager.go:414] "containerMap: removing container" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" Oct 30 01:42:00.652013 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.651956 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" Oct 30 01:42:00.652013 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.651977 5055 cpu_manager.go:414] "containerMap: removing container" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" Oct 30 01:42:00.652013 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.651982 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" Oct 30 01:42:00.652105 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.652073 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" Oct 30 01:42:00.652105 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.652086 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" containerName="istio-proxy" Oct 30 01:42:00.655362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.655342 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:00.680274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.680247 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts"] Oct 30 01:42:00.687704 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.687681 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ea0fc4\"" Oct 30 01:42:00.736372 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.736338 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsg7h\" (UniqueName: \"kubernetes.io/projected/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-kube-api-access-wsg7h\") pod \"payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts\" (UID: \"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:00.736659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.736638 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts\" (UID: \"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:00.851441 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.848594 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts\" (UID: \"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:00.851441 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.848668 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsg7h\" (UniqueName: \"kubernetes.io/projected/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-kube-api-access-wsg7h\") pod \"payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts\" (UID: \"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:00.852570 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.852514 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts\" (UID: \"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:00.867482 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.867451 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="316b9154-8e84-4e72-9250-d67591d60ed5" path="/var/lib/kubelet/pods/316b9154-8e84-4e72-9250-d67591d60ed5/volumes" Oct 30 01:42:00.879347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.876012 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsg7h\" (UniqueName: \"kubernetes.io/projected/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-kube-api-access-wsg7h\") pod \"payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts\" (UID: \"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:00.971113 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:00.970923 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:01.279462 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:01.279422 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts"] Oct 30 01:42:01.279923 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:42:01.279889 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb069f5fe_d89d_4fc8_a788_2d0cd7c77a05.slice/crio-2cad013d127fb2bba5f6353230d5662b0aa613e1bd908dce985b1997a8ff9180 WatchSource:0}: Error finding container 2cad013d127fb2bba5f6353230d5662b0aa613e1bd908dce985b1997a8ff9180: Status 404 returned error can't find the container with id 2cad013d127fb2bba5f6353230d5662b0aa613e1bd908dce985b1997a8ff9180 Oct 30 01:42:01.938077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:01.938044 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" event={"ID":"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05","Type":"ContainerStarted","Data":"f6dd641a03fc0d8da8c97b78496b272e7e52727fcb7d266ab7dae9b4d7cb8cb2"} Oct 30 01:42:01.938486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:01.938097 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" event={"ID":"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05","Type":"ContainerStarted","Data":"2cad013d127fb2bba5f6353230d5662b0aa613e1bd908dce985b1997a8ff9180"} Oct 30 01:42:01.938486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:01.938124 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:01.983020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:01.982966 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" podStartSLOduration=1.982949767 podStartE2EDuration="1.982949767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:01.978440137 +0000 UTC m=+1489.299397190" watchObservedRunningTime="2026-10-30 01:42:01.982949767 +0000 UTC m=+1489.303906809" Oct 30 01:42:05.937773 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:05.937744 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:42:12.947208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:12.947179 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:13.060081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.060051 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:42:13.063538 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.063514 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:42:13.442650 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.442609 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh"] Oct 30 01:42:13.448665 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.448642 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.483754 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.479974 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh"] Oct 30 01:42:13.487280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.487256 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-0ae258-openshift-default-dockercfg-9hvbg\"" Oct 30 01:42:13.536006 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.535970 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jjpw\" (UniqueName: \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-kube-api-access-4jjpw\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.536266 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.536241 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-certs\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.536560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.536540 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19a3a45-89c3-4f99-a84a-2645813b36d9-istiod-ca-cert\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.536702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.536686 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-socket\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.536814 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.536799 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-data\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.536930 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.536914 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-envoy\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.537063 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.537048 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-token\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.537205 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.537189 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-podinfo\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.537409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.537390 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-credential-socket\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.655348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.654440 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19a3a45-89c3-4f99-a84a-2645813b36d9-istiod-ca-cert\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.655348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.654513 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-socket\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.655348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.654540 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-data\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.655348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.654586 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-envoy\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.655348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.654637 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-token\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.655348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.654707 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-podinfo\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.655348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.654824 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-credential-socket\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.655348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.654913 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jjpw\" (UniqueName: \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-kube-api-access-4jjpw\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.655348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.654966 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-certs\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.655979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.655611 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-certs\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.670376 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.656476 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19a3a45-89c3-4f99-a84a-2645813b36d9-istiod-ca-cert\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.670376 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.657218 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-credential-socket\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.670376 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.657675 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-data\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.671104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.670980 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-socket\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.685521 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.678952 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-podinfo\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.711802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.708567 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-envoy\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.713765 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.712645 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-token\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.728059 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.727985 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jjpw\" (UniqueName: \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-kube-api-access-4jjpw\") pod \"e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:13.788579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:13.788550 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:14.173122 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:42:14.173091 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19a3a45_89c3_4f99_a84a_2645813b36d9.slice/crio-ac7248aeb2b99ece59675554cc53390accb242ad2d8c6e0f9c81ffce81ebe864 WatchSource:0}: Error finding container ac7248aeb2b99ece59675554cc53390accb242ad2d8c6e0f9c81ffce81ebe864: Status 404 returned error can't find the container with id ac7248aeb2b99ece59675554cc53390accb242ad2d8c6e0f9c81ffce81ebe864 Oct 30 01:42:14.173682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:14.173656 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh"] Oct 30 01:42:14.175927 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:14.175896 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:14.176028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:14.175957 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:14.176028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:14.175989 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:15.061424 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:15.061380 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" event={"ID":"f19a3a45-89c3-4f99-a84a-2645813b36d9","Type":"ContainerStarted","Data":"349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b"} Oct 30 01:42:15.061598 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:15.061440 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" event={"ID":"f19a3a45-89c3-4f99-a84a-2645813b36d9","Type":"ContainerStarted","Data":"ac7248aeb2b99ece59675554cc53390accb242ad2d8c6e0f9c81ffce81ebe864"} Oct 30 01:42:15.116989 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:15.116926 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" podStartSLOduration=2.1169081849999998 podStartE2EDuration="2.116908185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:15.115436077 +0000 UTC m=+1502.436393118" watchObservedRunningTime="2026-10-30 01:42:15.116908185 +0000 UTC m=+1502.437865225" Oct 30 01:42:15.790022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:15.789992 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:15.796202 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:15.796175 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:16.068203 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:16.068123 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:16.068979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:16.068961 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:42:18.040257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.040219 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn"] Oct 30 01:42:18.045132 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.045112 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.066801 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.066778 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ea0fc4-serving-cert\"" Oct 30 01:42:18.073823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.073797 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn"] Oct 30 01:42:18.128081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.128044 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.128264 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.128115 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-metrics-certs\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.128264 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.128226 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccfjr\" (UniqueName: \"kubernetes.io/projected/673cf0f4-cf51-4653-bc59-c79408b148ba-kube-api-access-ccfjr\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.128411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.128318 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673cf0f4-cf51-4653-bc59-c79408b148ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.243360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.232696 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.243360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.232835 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-metrics-certs\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.243360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.232931 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccfjr\" (UniqueName: \"kubernetes.io/projected/673cf0f4-cf51-4653-bc59-c79408b148ba-kube-api-access-ccfjr\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.243360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.232995 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673cf0f4-cf51-4653-bc59-c79408b148ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.243360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.234239 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673cf0f4-cf51-4653-bc59-c79408b148ba-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.243360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.239204 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.248110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.248046 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-metrics-certs\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.264550 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.264485 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccfjr\" (UniqueName: \"kubernetes.io/projected/673cf0f4-cf51-4653-bc59-c79408b148ba-kube-api-access-ccfjr\") pod \"maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.360860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.360772 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:18.674375 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:18.673958 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn"] Oct 30 01:42:19.096014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.095978 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" event={"ID":"673cf0f4-cf51-4653-bc59-c79408b148ba","Type":"ContainerStarted","Data":"8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964"} Oct 30 01:42:19.096505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.096035 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" event={"ID":"673cf0f4-cf51-4653-bc59-c79408b148ba","Type":"ContainerStarted","Data":"569dcd79c81e039bb55c3789d22a930323ae06321454db93e80fd71275e2f897"} Oct 30 01:42:19.096899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.096881 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:19.145381 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.145285 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" podStartSLOduration=0.887620669 podStartE2EDuration="1.145267037s" totalImagesPullingTime="257.646368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:19.143891248 +0000 UTC m=+1506.464848289" watchObservedRunningTime="2026-10-30 01:42:19.145267037 +0000 UTC m=+1506.466224079" Oct 30 01:42:19.236374 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:42:19.233491 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:42:19.877256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.877226 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf"] Oct 30 01:42:19.881719 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.881682 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:19.916418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.916391 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf"] Oct 30 01:42:19.916714 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.916693 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-c22e6b-serving-cert\"" Oct 30 01:42:19.983148 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.983113 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkdwj\" (UniqueName: \"kubernetes.io/projected/7ff14eef-46bf-4fad-88b6-06d591dfc033-kube-api-access-lkdwj\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:19.983310 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.983168 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff14eef-46bf-4fad-88b6-06d591dfc033-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:19.983310 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.983276 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-metrics-certs\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:19.983468 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:19.983401 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:20.084809 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:20.084770 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkdwj\" (UniqueName: \"kubernetes.io/projected/7ff14eef-46bf-4fad-88b6-06d591dfc033-kube-api-access-lkdwj\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:20.085020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:20.084846 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff14eef-46bf-4fad-88b6-06d591dfc033-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:20.085020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:20.084912 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-metrics-certs\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:20.085020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:20.084980 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:20.086652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:20.086621 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff14eef-46bf-4fad-88b6-06d591dfc033-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:20.098910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:20.098878 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-metrics-certs\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:20.101158 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:20.101128 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-maas-api-tls\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:20.106802 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:20.106770 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkdwj\" (UniqueName: \"kubernetes.io/projected/7ff14eef-46bf-4fad-88b6-06d591dfc033-kube-api-access-lkdwj\") pod \"maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:20.197254 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:20.197162 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:20.502096 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:42:20.502056 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff14eef_46bf_4fad_88b6_06d591dfc033.slice/crio-d1b28392320865b03673e0a880ab56200edea1ac800eb720ff621d4a7dc149c5 WatchSource:0}: Error finding container d1b28392320865b03673e0a880ab56200edea1ac800eb720ff621d4a7dc149c5: Status 404 returned error can't find the container with id d1b28392320865b03673e0a880ab56200edea1ac800eb720ff621d4a7dc149c5 Oct 30 01:42:20.503465 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:20.503436 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf"] Oct 30 01:42:21.136375 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:21.136244 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" event={"ID":"7ff14eef-46bf-4fad-88b6-06d591dfc033","Type":"ContainerStarted","Data":"98fe2c45206fa3904ec3c876616c09a5a0cd594f2417ab904570b6c7571f717f"} Oct 30 01:42:21.136375 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:21.136298 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" event={"ID":"7ff14eef-46bf-4fad-88b6-06d591dfc033","Type":"ContainerStarted","Data":"d1b28392320865b03673e0a880ab56200edea1ac800eb720ff621d4a7dc149c5"} Oct 30 01:42:21.136375 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:21.136361 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:21.176944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:21.176888 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" podStartSLOduration=1.852641075 podStartE2EDuration="2.176872145s" totalImagesPullingTime="324.23107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:21.175140341 +0000 UTC m=+1508.496097383" watchObservedRunningTime="2026-10-30 01:42:21.176872145 +0000 UTC m=+1508.497829206" Oct 30 01:42:26.141636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:26.141607 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:42:28.155831 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:28.155799 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:42:34.773760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:34.773718 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6"] Oct 30 01:42:34.774262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:34.773945 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" containerID="cri-o://1e6813fa5a0dc96131839cbf0265a63e3e7f339f55130bb7b7569dcf5d2d9ead" gracePeriod=30 Oct 30 01:42:40.331826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.331721 5055 generic.go:356] "Generic (PLEG): container finished" podID="222113ca-8659-476a-97a0-891ed690141e" containerID="1e6813fa5a0dc96131839cbf0265a63e3e7f339f55130bb7b7569dcf5d2d9ead" exitCode=0 Oct 30 01:42:40.331826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.331749 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" event={"ID":"222113ca-8659-476a-97a0-891ed690141e","Type":"ContainerDied","Data":"1e6813fa5a0dc96131839cbf0265a63e3e7f339f55130bb7b7569dcf5d2d9ead"} Oct 30 01:42:40.332258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.331853 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" event={"ID":"222113ca-8659-476a-97a0-891ed690141e","Type":"ContainerDied","Data":"e8bbd309e27f11e17759878339da6307e1fa00c472efbd386b637d9865880a61"} Oct 30 01:42:40.332258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.331897 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8bbd309e27f11e17759878339da6307e1fa00c472efbd386b637d9865880a61" Oct 30 01:42:40.334238 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.334222 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:42:40.382347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.361383 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-credential-socket\") pod \"222113ca-8659-476a-97a0-891ed690141e\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " Oct 30 01:42:40.382347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.361432 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/222113ca-8659-476a-97a0-891ed690141e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222113ca-8659-476a-97a0-891ed690141e-istiod-ca-cert\") pod \"222113ca-8659-476a-97a0-891ed690141e\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " Oct 30 01:42:40.382347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.361463 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/222113ca-8659-476a-97a0-891ed690141e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222113ca-8659-476a-97a0-891ed690141e-istio-podinfo\") pod \"222113ca-8659-476a-97a0-891ed690141e\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " Oct 30 01:42:40.382347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.361509 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-kube-api-access-vd758\" (UniqueName: \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-kube-api-access-vd758\") pod \"222113ca-8659-476a-97a0-891ed690141e\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " Oct 30 01:42:40.382347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.361583 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-socket\") pod \"222113ca-8659-476a-97a0-891ed690141e\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " Oct 30 01:42:40.382347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.361684 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-certs\") pod \"222113ca-8659-476a-97a0-891ed690141e\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " Oct 30 01:42:40.382347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.361770 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-istio-token\" (UniqueName: \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-istio-token\") pod \"222113ca-8659-476a-97a0-891ed690141e\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " Oct 30 01:42:40.382347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.373473 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-envoy\") pod \"222113ca-8659-476a-97a0-891ed690141e\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " Oct 30 01:42:40.382347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.373652 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-data\") pod \"222113ca-8659-476a-97a0-891ed690141e\" (UID: \"222113ca-8659-476a-97a0-891ed690141e\") " Oct 30 01:42:40.392359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.385499 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222113ca-8659-476a-97a0-891ed690141e-istiod-ca-cert" pod "222113ca-8659-476a-97a0-891ed690141e" (UID: "222113ca-8659-476a-97a0-891ed690141e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:40.392359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.385870 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-data" pod "222113ca-8659-476a-97a0-891ed690141e" (UID: "222113ca-8659-476a-97a0-891ed690141e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:40.396425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.396391 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-socket" pod "222113ca-8659-476a-97a0-891ed690141e" (UID: "222113ca-8659-476a-97a0-891ed690141e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:40.399390 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.398040 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-credential-socket" pod "222113ca-8659-476a-97a0-891ed690141e" (UID: "222113ca-8659-476a-97a0-891ed690141e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:40.399874 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.399844 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-kube-api-access-vd758" pod "222113ca-8659-476a-97a0-891ed690141e" (UID: "222113ca-8659-476a-97a0-891ed690141e"). InnerVolumeSpecName "kube-api-access-vd758". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:40.432374 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.407216 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-istio-token" pod "222113ca-8659-476a-97a0-891ed690141e" (UID: "222113ca-8659-476a-97a0-891ed690141e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:40.432374 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.409713 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/222113ca-8659-476a-97a0-891ed690141e-istio-podinfo" pod "222113ca-8659-476a-97a0-891ed690141e" (UID: "222113ca-8659-476a-97a0-891ed690141e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:42:40.438433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.437698 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-envoy" pod "222113ca-8659-476a-97a0-891ed690141e" (UID: "222113ca-8659-476a-97a0-891ed690141e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:40.438433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.437949 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-certs" pod "222113ca-8659-476a-97a0-891ed690141e" (UID: "222113ca-8659-476a-97a0-891ed690141e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:40.490443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.486409 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:40.490443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.486446 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/222113ca-8659-476a-97a0-891ed690141e-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:40.490443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.486460 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/222113ca-8659-476a-97a0-891ed690141e-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:40.490443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.486475 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd758\" (UniqueName: \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-kube-api-access-vd758\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:40.490443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.486490 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:40.490443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.486503 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:40.490443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.486517 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/222113ca-8659-476a-97a0-891ed690141e-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:40.490443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.486529 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:40.490443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:40.486542 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/222113ca-8659-476a-97a0-891ed690141e-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:41.339504 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:41.339468 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6" Oct 30 01:42:41.427470 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:41.427440 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6"] Oct 30 01:42:41.430997 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:41.430855 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-spvp6"] Oct 30 01:42:42.225440 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.225406 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh"] Oct 30 01:42:42.226657 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.226630 5055 cpu_manager.go:414] "containerMap: removing container" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" Oct 30 01:42:42.226821 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.226807 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" Oct 30 01:42:42.227057 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.227042 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="222113ca-8659-476a-97a0-891ed690141e" containerName="istio-proxy" Oct 30 01:42:42.233196 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.233172 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh"] Oct 30 01:42:42.233397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.233306 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.249732 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.249718 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-0ae258-serving-cert\"" Oct 30 01:42:42.336389 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.336355 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-metrics-certs\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.336686 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.336666 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ac42ae-8948-4f15-ab7a-66717bb47fc2-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.336914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.336842 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.337147 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.337127 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxkmt\" (UniqueName: \"kubernetes.io/projected/80ac42ae-8948-4f15-ab7a-66717bb47fc2-kube-api-access-hxkmt\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.440662 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.440614 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-metrics-certs\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.441290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.441265 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ac42ae-8948-4f15-ab7a-66717bb47fc2-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.441480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.441463 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.441722 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.441703 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxkmt\" (UniqueName: \"kubernetes.io/projected/80ac42ae-8948-4f15-ab7a-66717bb47fc2-kube-api-access-hxkmt\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.445418 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:42:42.442965 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-serving-cert: secret "maas-api-e2e-praxis-a-0ae258-serving-cert" not found Oct 30 01:42:42.445630 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:42:42.445615 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-maas-api-tls podName:80ac42ae-8948-4f15-ab7a-66717bb47fc2 nodeName:}" failed. No retries permitted until 2026-10-30 01:42:42.945589994 +0000 UTC m=+1530.266547032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-maas-api-tls") pod "maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" (UID: "80ac42ae-8948-4f15-ab7a-66717bb47fc2") : secret "maas-api-e2e-praxis-a-0ae258-serving-cert" not found Oct 30 01:42:42.446690 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.446668 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ac42ae-8948-4f15-ab7a-66717bb47fc2-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.461897 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.461872 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-metrics-certs\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.491402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.491036 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxkmt\" (UniqueName: \"kubernetes.io/projected/80ac42ae-8948-4f15-ab7a-66717bb47fc2-kube-api-access-hxkmt\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.835836 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.835760 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="222113ca-8659-476a-97a0-891ed690141e" path="/var/lib/kubelet/pods/222113ca-8659-476a-97a0-891ed690141e/volumes" Oct 30 01:42:42.949513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.949464 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:42.952083 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:42.952056 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-maas-api-tls\") pod \"maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:43.213042 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:43.212996 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:43.659821 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:43.659747 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5"] Oct 30 01:42:43.660173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:43.660000 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerName="istio-proxy" containerID="cri-o://06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95" gracePeriod=30 Oct 30 01:42:43.706394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:43.705978 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh"] Oct 30 01:42:43.707940 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:42:43.707885 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ac42ae_8948_4f15_ab7a_66717bb47fc2.slice/crio-00b51e618ba5ee878af81e3f355e2642f3e53e615e4252ebc42ff4c7e6fe437f WatchSource:0}: Error finding container 00b51e618ba5ee878af81e3f355e2642f3e53e615e4252ebc42ff4c7e6fe437f: Status 404 returned error can't find the container with id 00b51e618ba5ee878af81e3f355e2642f3e53e615e4252ebc42ff4c7e6fe437f Oct 30 01:42:44.389235 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:44.389146 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" event={"ID":"80ac42ae-8948-4f15-ab7a-66717bb47fc2","Type":"ContainerStarted","Data":"0330cd61df61478418012e85c32bac39ccbdd4dd51da5a61941a1d78fdf6a333"} Oct 30 01:42:44.389235 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:44.389214 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" event={"ID":"80ac42ae-8948-4f15-ab7a-66717bb47fc2","Type":"ContainerStarted","Data":"00b51e618ba5ee878af81e3f355e2642f3e53e615e4252ebc42ff4c7e6fe437f"} Oct 30 01:42:44.392365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:44.390418 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:44.433561 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:44.433373 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" podStartSLOduration=2.075750055 podStartE2EDuration="2.43335499s" totalImagesPullingTime="357.604935ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:44.43257754 +0000 UTC m=+1531.753534580" watchObservedRunningTime="2026-10-30 01:42:44.43335499 +0000 UTC m=+1531.754312030" Oct 30 01:42:47.417050 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:47.417014 5055 generic.go:356] "Generic (PLEG): container finished" podID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerID="1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39" exitCode=0 Oct 30 01:42:47.417645 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:47.417099 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" event={"ID":"c94cdcb7-32d4-437f-af1f-df2a30c79d3b","Type":"ContainerDied","Data":"1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39"} Oct 30 01:42:47.417645 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:47.417600 5055 scope.go:122] "RemoveContainer" containerID="1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39" Oct 30 01:42:47.419219 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:47.419188 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:47.419305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:47.419258 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:47.419305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:47.419290 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:48.425304 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:48.425190 5055 generic.go:356] "Generic (PLEG): container finished" podID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerID="9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad" exitCode=0 Oct 30 01:42:48.425304 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:48.425266 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" event={"ID":"1bafa687-8080-4ee5-9388-fb0d6d74ffc1","Type":"ContainerDied","Data":"9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad"} Oct 30 01:42:48.425887 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:48.425770 5055 scope.go:122] "RemoveContainer" containerID="9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad" Oct 30 01:42:48.427505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:48.427474 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:48.427625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:48.427563 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:48.427625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:48.427576 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" event={"ID":"c94cdcb7-32d4-437f-af1f-df2a30c79d3b","Type":"ContainerStarted","Data":"0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6"} Oct 30 01:42:48.427625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:48.427615 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:48.427899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:48.427881 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:42:48.428071 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:48.428048 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 01:42:48.428156 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:48.428089 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 01:42:49.058062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.058040 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:42:49.219202 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.219166 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-kube-api-access-j6x8v\" (UniqueName: \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-kube-api-access-j6x8v\") pod \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " Oct 30 01:42:49.219671 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.219645 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-envoy\") pod \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " Oct 30 01:42:49.219913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.219896 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-socket\") pod \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " Oct 30 01:42:49.220047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.220037 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istiod-ca-cert\") pod \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " Oct 30 01:42:49.220161 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.220149 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-podinfo\") pod \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " Oct 30 01:42:49.220283 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.220123 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-socket" pod "1bd262c4-5d37-4cd2-85fe-7e3bbb053435" (UID: "1bd262c4-5d37-4cd2-85fe-7e3bbb053435"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:49.220405 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.220395 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-data\") pod \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " Oct 30 01:42:49.220518 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.220504 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-token\" (UniqueName: \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-token\") pod \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " Oct 30 01:42:49.220653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.220638 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-certs\") pod \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " Oct 30 01:42:49.220783 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.220772 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-credential-socket\") pod \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\" (UID: \"1bd262c4-5d37-4cd2-85fe-7e3bbb053435\") " Oct 30 01:42:49.220963 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.220503 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istiod-ca-cert" pod "1bd262c4-5d37-4cd2-85fe-7e3bbb053435" (UID: "1bd262c4-5d37-4cd2-85fe-7e3bbb053435"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:49.221035 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.220702 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-data" pod "1bd262c4-5d37-4cd2-85fe-7e3bbb053435" (UID: "1bd262c4-5d37-4cd2-85fe-7e3bbb053435"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:49.221035 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.220872 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-certs" pod "1bd262c4-5d37-4cd2-85fe-7e3bbb053435" (UID: "1bd262c4-5d37-4cd2-85fe-7e3bbb053435"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:49.221127 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.221030 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-credential-socket" pod "1bd262c4-5d37-4cd2-85fe-7e3bbb053435" (UID: "1bd262c4-5d37-4cd2-85fe-7e3bbb053435"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:49.221623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.221605 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:49.221743 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.221733 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:49.221812 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.221804 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:49.221873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.221864 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:49.221927 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.221921 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:49.222099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.222066 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-kube-api-access-j6x8v" pod "1bd262c4-5d37-4cd2-85fe-7e3bbb053435" (UID: "1bd262c4-5d37-4cd2-85fe-7e3bbb053435"). InnerVolumeSpecName "kube-api-access-j6x8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:49.222664 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.222637 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-podinfo" pod "1bd262c4-5d37-4cd2-85fe-7e3bbb053435" (UID: "1bd262c4-5d37-4cd2-85fe-7e3bbb053435"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:42:49.222924 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.222903 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-envoy" pod "1bd262c4-5d37-4cd2-85fe-7e3bbb053435" (UID: "1bd262c4-5d37-4cd2-85fe-7e3bbb053435"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:49.223121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.223105 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-token" pod "1bd262c4-5d37-4cd2-85fe-7e3bbb053435" (UID: "1bd262c4-5d37-4cd2-85fe-7e3bbb053435"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:49.241672 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:42:49.241637 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:42:49.323427 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.323345 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:49.323427 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.323378 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:49.323427 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.323392 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6x8v\" (UniqueName: \"kubernetes.io/projected/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-kube-api-access-j6x8v\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:49.323427 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.323406 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bd262c4-5d37-4cd2-85fe-7e3bbb053435-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:49.390352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.387191 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts"] Oct 30 01:42:49.428157 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.428121 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 01:42:49.428688 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.428177 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 01:42:49.439807 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.439770 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" event={"ID":"1bafa687-8080-4ee5-9388-fb0d6d74ffc1","Type":"ContainerStarted","Data":"1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1"} Oct 30 01:42:49.441459 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.441427 5055 generic.go:356] "Generic (PLEG): container finished" podID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerID="06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95" exitCode=0 Oct 30 01:42:49.441737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.441481 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" Oct 30 01:42:49.441737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.441508 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" event={"ID":"1bd262c4-5d37-4cd2-85fe-7e3bbb053435","Type":"ContainerDied","Data":"06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95"} Oct 30 01:42:49.441737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.441560 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5" event={"ID":"1bd262c4-5d37-4cd2-85fe-7e3bbb053435","Type":"ContainerDied","Data":"cf0212fa38072fefe32d997696477dcd647f0a81d36f21c9911ca466cb6091b4"} Oct 30 01:42:49.441737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.441586 5055 scope.go:122] "RemoveContainer" containerID="06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95" Oct 30 01:42:49.472494 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.470934 5055 scope.go:122] "RemoveContainer" containerID="06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95" Oct 30 01:42:49.472494 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:42:49.471310 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95\": container with ID starting with 06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95 not found: ID does not exist" containerID="06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95" Oct 30 01:42:49.472494 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.471365 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95"} err="failed to get container status \"06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95\": rpc error: code = NotFound desc = could not find container \"06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95\": container with ID starting with 06ce03e9b0377f7d12e38faa0cfb5071ee7e4f38534c6fa59195c1d7771cae95 not found: ID does not exist" Oct 30 01:42:49.628407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.627607 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5"] Oct 30 01:42:49.660180 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.660139 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-84xp5"] Oct 30 01:42:49.936409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.934290 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:42:49.936409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.934446 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:42:49.936409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:49.934501 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 01:42:50.431361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:50.428486 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:42:50.431361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:50.429246 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 01:42:50.431361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:50.429294 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 01:42:50.837395 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:50.837359 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" path="/var/lib/kubelet/pods/1bd262c4-5d37-4cd2-85fe-7e3bbb053435/volumes" Oct 30 01:42:50.930910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:50.930652 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:42:50.930910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:50.930710 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 01:42:51.125047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.124961 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb"] Oct 30 01:42:51.137351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.126252 5055 cpu_manager.go:414] "containerMap: removing container" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerName="istio-proxy" Oct 30 01:42:51.139230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.139205 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerName="istio-proxy" Oct 30 01:42:51.139681 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.139666 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd262c4-5d37-4cd2-85fe-7e3bbb053435" containerName="istio-proxy" Oct 30 01:42:51.230486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.230450 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb"] Oct 30 01:42:51.237793 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.230815 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.298413 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.280635 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-podinfo\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.379420 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.318486 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-socket\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.401350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.396119 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-envoy\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.401667 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.401638 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-credential-socket\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.446153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.446123 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c75d79-a532-4de1-9d1e-12925e33c5dd-istiod-ca-cert\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.446689 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.446668 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-data\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.446868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.446853 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-token\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.447056 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.447038 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bfzq\" (UniqueName: \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-kube-api-access-2bfzq\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.447209 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.447195 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-certs\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.447483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.436248 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 01:42:51.447635 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.447611 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 01:42:51.447804 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.382669 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-be8a7d-openshift-default-dockercfg-mpg4g\"" Oct 30 01:42:51.553177 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.553131 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-socket\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.553931 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.553913 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-envoy\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.554417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.554400 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-credential-socket\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.554585 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.554571 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c75d79-a532-4de1-9d1e-12925e33c5dd-istiod-ca-cert\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.554700 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.554684 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-data\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.554850 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.554834 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-token\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.555007 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.554993 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bfzq\" (UniqueName: \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-kube-api-access-2bfzq\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.555134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.555123 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-certs\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.555298 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.555282 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-podinfo\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.555508 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.555343 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c75d79-a532-4de1-9d1e-12925e33c5dd-istiod-ca-cert\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.555593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.553824 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-socket\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.556257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.555870 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-data\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.556257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.554734 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-credential-socket\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.556257 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.556114 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-certs\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.572359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.570628 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-envoy\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.615352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.609347 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-podinfo\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.624351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.619816 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bfzq\" (UniqueName: \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-kube-api-access-2bfzq\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.624512 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.624445 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-token\") pod \"e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.836350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.834576 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:51.953354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.953067 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:42:51.953354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:51.953131 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 01:42:52.430234 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:52.430192 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 01:42:52.430439 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:52.430255 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 01:42:52.450349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:52.438019 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb"] Oct 30 01:42:52.453834 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:52.453783 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:52.453953 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:52.453888 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:52.453953 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:52.453935 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:42:52.555587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:52.555561 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" event={"ID":"59c75d79-a532-4de1-9d1e-12925e33c5dd","Type":"ContainerStarted","Data":"cdf91b85038c0c7a3f74d9906d6928071d15ab7727eef100600e5cbfacd20318"} Oct 30 01:42:52.946495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:52.946465 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:42:52.946784 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:52.946759 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 01:42:53.432677 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:53.432639 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Oct 30 01:42:53.432953 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:53.432922 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Oct 30 01:42:53.562637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:53.562606 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" event={"ID":"59c75d79-a532-4de1-9d1e-12925e33c5dd","Type":"ContainerStarted","Data":"c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654"} Oct 30 01:42:53.641943 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:53.641892 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" podStartSLOduration=2.641872117 podStartE2EDuration="2.641872117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:53.627264605 +0000 UTC m=+1540.948221643" watchObservedRunningTime="2026-10-30 01:42:53.641872117 +0000 UTC m=+1540.962829157" Oct 30 01:42:53.842359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:53.840393 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:53.907721 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:53.907682 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:53.939351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:53.939301 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:42:53.943442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:53.943395 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 01:42:54.388752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.388712 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" podUID="b069f5fe-d89d-4fc8-a788-2d0cd7c77a05" containerName="payload-processing" containerID="cri-o://f6dd641a03fc0d8da8c97b78496b272e7e52727fcb7d266ab7dae9b4d7cb8cb2" gracePeriod=25 Oct 30 01:42:54.591697 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.591661 5055 generic.go:356] "Generic (PLEG): container finished" podID="b069f5fe-d89d-4fc8-a788-2d0cd7c77a05" containerID="f6dd641a03fc0d8da8c97b78496b272e7e52727fcb7d266ab7dae9b4d7cb8cb2" exitCode=0 Oct 30 01:42:54.603898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.603862 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" event={"ID":"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05","Type":"ContainerDied","Data":"f6dd641a03fc0d8da8c97b78496b272e7e52727fcb7d266ab7dae9b4d7cb8cb2"} Oct 30 01:42:54.604170 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.604154 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:54.616519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.616485 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:42:54.752378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.751306 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:42:54.753272 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.753153 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:42:54.755765 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.755746 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:42:54.847285 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.847093 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:54.930696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.930662 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Oct 30 01:42:54.930886 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.930748 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Oct 30 01:42:54.936445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.936423 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-kube-api-access-wsg7h\" (UniqueName: \"kubernetes.io/projected/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-kube-api-access-wsg7h\") pod \"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05\" (UID: \"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05\") " Oct 30 01:42:54.936570 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.936520 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-plugins-config-volume\") pod \"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05\" (UID: \"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05\") " Oct 30 01:42:54.937065 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.937015 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-plugins-config-volume" pod "b069f5fe-d89d-4fc8-a788-2d0cd7c77a05" (UID: "b069f5fe-d89d-4fc8-a788-2d0cd7c77a05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:54.938892 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:54.938861 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-kube-api-access-wsg7h" pod "b069f5fe-d89d-4fc8-a788-2d0cd7c77a05" (UID: "b069f5fe-d89d-4fc8-a788-2d0cd7c77a05"). InnerVolumeSpecName "kube-api-access-wsg7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:55.039303 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:55.039216 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsg7h\" (UniqueName: \"kubernetes.io/projected/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-kube-api-access-wsg7h\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:55.039303 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:55.039253 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05-plugins-config-volume\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:42:55.602030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:55.601996 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" Oct 30 01:42:55.602030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:55.602018 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts" event={"ID":"b069f5fe-d89d-4fc8-a788-2d0cd7c77a05","Type":"ContainerDied","Data":"2cad013d127fb2bba5f6353230d5662b0aa613e1bd908dce985b1997a8ff9180"} Oct 30 01:42:55.602599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:55.602083 5055 scope.go:122] "RemoveContainer" containerID="f6dd641a03fc0d8da8c97b78496b272e7e52727fcb7d266ab7dae9b4d7cb8cb2" Oct 30 01:42:55.662693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:55.662664 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts"] Oct 30 01:42:55.677062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:55.677034 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea0fc4-84978c6cf6-7lmts"] Oct 30 01:42:55.935677 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:55.935600 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:42:55.935974 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:55.935952 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:42:55.936638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:55.936623 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:42:56.001663 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.001627 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz"] Oct 30 01:42:56.002237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.002221 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b069f5fe-d89d-4fc8-a788-2d0cd7c77a05" containerName="payload-processing" Oct 30 01:42:56.002237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.002237 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b069f5fe-d89d-4fc8-a788-2d0cd7c77a05" containerName="payload-processing" Oct 30 01:42:56.002467 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.002448 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b069f5fe-d89d-4fc8-a788-2d0cd7c77a05" containerName="payload-processing" Oct 30 01:42:56.005835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.005812 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.034191 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.034173 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-c9d174-serving-cert\"" Oct 30 01:42:56.039438 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.039404 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz"] Oct 30 01:42:56.059391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.056118 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtdf\" (UniqueName: \"kubernetes.io/projected/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-kube-api-access-ndtdf\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.059391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.056213 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.059391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.056274 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-metrics-certs\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.059391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.057122 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.159133 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.159104 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.159321 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.159220 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtdf\" (UniqueName: \"kubernetes.io/projected/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-kube-api-access-ndtdf\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.159321 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:42:56.159267 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-serving-cert: secret "maas-api-e2e-praxis-b-c9d174-serving-cert" not found Oct 30 01:42:56.159321 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:42:56.159351 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-maas-api-tls podName:bfb2f44c-9d01-496e-9d63-9e79cc2c2918 nodeName:}" failed. No retries permitted until 2026-10-30 01:42:56.659316125 +0000 UTC m=+1543.980273156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-maas-api-tls") pod "maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" (UID: "bfb2f44c-9d01-496e-9d63-9e79cc2c2918") : secret "maas-api-e2e-praxis-b-c9d174-serving-cert" not found Oct 30 01:42:56.159557 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.159273 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.159557 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.159438 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-metrics-certs\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.159852 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.159835 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.161970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.161945 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-metrics-certs\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.181917 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.181895 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtdf\" (UniqueName: \"kubernetes.io/projected/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-kube-api-access-ndtdf\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.667209 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.667163 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.670267 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.670238 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-maas-api-tls\") pod \"maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:56.821065 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.821032 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b069f5fe-d89d-4fc8-a788-2d0cd7c77a05" path="/var/lib/kubelet/pods/b069f5fe-d89d-4fc8-a788-2d0cd7c77a05/volumes" Oct 30 01:42:56.921719 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:56.921629 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:57.342981 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:42:57.342949 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfb2f44c_9d01_496e_9d63_9e79cc2c2918.slice/crio-ff1fd84c7bd4715f5fb4defe3a55d52d1daa16be118d7a96bb329345f6edc3b4 WatchSource:0}: Error finding container ff1fd84c7bd4715f5fb4defe3a55d52d1daa16be118d7a96bb329345f6edc3b4: Status 404 returned error can't find the container with id ff1fd84c7bd4715f5fb4defe3a55d52d1daa16be118d7a96bb329345f6edc3b4 Oct 30 01:42:57.344413 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:57.344388 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz"] Oct 30 01:42:57.627944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:57.627852 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" event={"ID":"bfb2f44c-9d01-496e-9d63-9e79cc2c2918","Type":"ContainerStarted","Data":"ff1fd84c7bd4715f5fb4defe3a55d52d1daa16be118d7a96bb329345f6edc3b4"} Oct 30 01:42:58.636145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:58.636103 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" event={"ID":"bfb2f44c-9d01-496e-9d63-9e79cc2c2918","Type":"ContainerStarted","Data":"57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9"} Oct 30 01:42:58.636562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:58.636261 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:42:58.684361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:42:58.683763 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" podStartSLOduration=3.2895211 podStartE2EDuration="3.683743019s" totalImagesPullingTime="394.221919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:58.680771945 +0000 UTC m=+1546.001728986" watchObservedRunningTime="2026-10-30 01:42:58.683743019 +0000 UTC m=+1546.004700059" Oct 30 01:43:04.655992 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:04.655913 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:43:06.704140 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:06.704112 5055 generic.go:356] "Generic (PLEG): container finished" podID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerID="349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b" exitCode=0 Oct 30 01:43:06.704590 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:06.704171 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" event={"ID":"f19a3a45-89c3-4f99-a84a-2645813b36d9","Type":"ContainerDied","Data":"349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b"} Oct 30 01:43:06.704638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:06.704625 5055 scope.go:122] "RemoveContainer" containerID="349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b" Oct 30 01:43:06.706227 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:06.706197 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:43:06.706355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:06.706262 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:43:06.706355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:06.706293 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:43:07.712411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:07.712373 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" event={"ID":"f19a3a45-89c3-4f99-a84a-2645813b36d9","Type":"ContainerStarted","Data":"dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd"} Oct 30 01:43:07.794352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:07.793912 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:43:07.794550 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:07.794408 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:43:07.794550 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:07.794460 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 01:43:08.789856 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:08.789822 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:43:08.790476 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:08.790441 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 01:43:09.790258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:09.790224 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:43:09.790761 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:09.790286 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 01:43:10.790145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:10.790108 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:43:10.790383 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:10.790163 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 01:43:11.793524 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:11.793487 5055 patch_prober.go:29] interesting pod/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 01:43:11.794117 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:11.794086 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 01:43:13.073889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:13.073454 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:43:13.087265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:13.084411 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:43:13.110510 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:13.110480 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:43:14.092921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:14.092886 5055 scope.go:122] "RemoveContainer" containerID="1e6813fa5a0dc96131839cbf0265a63e3e7f339f55130bb7b7569dcf5d2d9ead" Oct 30 01:43:15.274074 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.274033 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr"] Oct 30 01:43:15.298063 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.298024 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.325041 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.323435 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr"] Oct 30 01:43:15.328812 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.328780 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-58632b-openshift-default-dockercfg-vd59h\"" Oct 30 01:43:15.561312 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.561205 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-socket\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.564060 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.564019 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-certs\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.567559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.567522 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-podinfo\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.567830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.567815 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-envoy\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.567955 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.567940 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t5cz\" (UniqueName: \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-kube-api-access-2t5cz\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.568073 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.568061 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-data\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.568175 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.568163 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-credential-socket\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.568384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.568370 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-token\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.568499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.568487 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae8f195-ce05-4880-8cc0-a76799258ff0-istiod-ca-cert\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.670234 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.670198 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-socket\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.670648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.670625 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-certs\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.670853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.670836 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-podinfo\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.671020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.671004 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-envoy\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.671145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.671129 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t5cz\" (UniqueName: \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-kube-api-access-2t5cz\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.671263 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.670671 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-socket\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.671377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.670892 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-certs\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.671468 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.671429 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-data\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.671585 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.671570 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-credential-socket\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.671695 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.671678 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-data\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.671803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.671759 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-token\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.671920 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.671904 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae8f195-ce05-4880-8cc0-a76799258ff0-istiod-ca-cert\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.672345 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.672302 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-credential-socket\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.672798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.672782 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae8f195-ce05-4880-8cc0-a76799258ff0-istiod-ca-cert\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.677929 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.677908 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-envoy\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.678932 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.678892 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-podinfo\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.694406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.694237 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-token\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.694406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.694407 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t5cz\" (UniqueName: \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-kube-api-access-2t5cz\") pod \"e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:15.932223 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:15.932120 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:16.270455 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:43:16.268477 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae8f195_ce05_4880_8cc0_a76799258ff0.slice/crio-5151dfac0bf01489d20d9fd84fd6a362d22dce78db543f8785d4e164104f9db8 WatchSource:0}: Error finding container 5151dfac0bf01489d20d9fd84fd6a362d22dce78db543f8785d4e164104f9db8: Status 404 returned error can't find the container with id 5151dfac0bf01489d20d9fd84fd6a362d22dce78db543f8785d4e164104f9db8 Oct 30 01:43:16.270455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:16.270267 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr"] Oct 30 01:43:16.272192 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:16.272154 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:43:16.272270 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:16.272248 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:43:16.272308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:16.272292 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:43:16.800197 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:16.800161 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" event={"ID":"6ae8f195-ce05-4880-8cc0-a76799258ff0","Type":"ContainerStarted","Data":"561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8"} Oct 30 01:43:16.800674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:16.800229 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" event={"ID":"6ae8f195-ce05-4880-8cc0-a76799258ff0","Type":"ContainerStarted","Data":"5151dfac0bf01489d20d9fd84fd6a362d22dce78db543f8785d4e164104f9db8"} Oct 30 01:43:16.868981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:16.868913 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" podStartSLOduration=1.86889398 podStartE2EDuration="1.86889398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:16.854244558 +0000 UTC m=+1564.175201599" watchObservedRunningTime="2026-10-30 01:43:16.86889398 +0000 UTC m=+1564.189851021" Oct 30 01:43:16.933180 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:16.933150 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:17.940856 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:17.940828 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:18.823310 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:18.823277 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:18.823549 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:18.823389 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:43:19.229755 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:43:19.229723 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:43:23.728218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:23.728133 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb"] Oct 30 01:43:23.728718 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:23.728623 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" podUID="d9380eb7-166e-4602-a7c5-83b8b9e0d84f" containerName="maas-api" containerID="cri-o://df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e" gracePeriod=30 Oct 30 01:43:24.380390 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.380355 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:43:24.449876 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.449844 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-kube-api-access-bvwhh\" (UniqueName: \"kubernetes.io/projected/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-kube-api-access-bvwhh\") pod \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " Oct 30 01:43:24.450040 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.449958 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-openshift-service-ca\") pod \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " Oct 30 01:43:24.450088 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.450076 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-maas-api-tls\") pod \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " Oct 30 01:43:24.450178 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.450158 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-metrics-certs\") pod \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\" (UID: \"d9380eb7-166e-4602-a7c5-83b8b9e0d84f\") " Oct 30 01:43:24.450532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.450500 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-openshift-service-ca" pod "d9380eb7-166e-4602-a7c5-83b8b9e0d84f" (UID: "d9380eb7-166e-4602-a7c5-83b8b9e0d84f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:24.450828 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.450792 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:43:24.452607 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.452578 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-metrics-certs" pod "d9380eb7-166e-4602-a7c5-83b8b9e0d84f" (UID: "d9380eb7-166e-4602-a7c5-83b8b9e0d84f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:24.452810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.452790 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-kube-api-access-bvwhh" pod "d9380eb7-166e-4602-a7c5-83b8b9e0d84f" (UID: "d9380eb7-166e-4602-a7c5-83b8b9e0d84f"). InnerVolumeSpecName "kube-api-access-bvwhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:24.452896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.452875 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-maas-api-tls" pod "d9380eb7-166e-4602-a7c5-83b8b9e0d84f" (UID: "d9380eb7-166e-4602-a7c5-83b8b9e0d84f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:24.552112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.552028 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvwhh\" (UniqueName: \"kubernetes.io/projected/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-kube-api-access-bvwhh\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:43:24.552112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.552067 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:43:24.552112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.552081 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9380eb7-166e-4602-a7c5-83b8b9e0d84f-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:43:24.857283 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.857250 5055 generic.go:356] "Generic (PLEG): container finished" podID="d9380eb7-166e-4602-a7c5-83b8b9e0d84f" containerID="df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e" exitCode=0 Oct 30 01:43:24.857690 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.857298 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" Oct 30 01:43:24.857690 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.857354 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" event={"ID":"d9380eb7-166e-4602-a7c5-83b8b9e0d84f","Type":"ContainerDied","Data":"df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e"} Oct 30 01:43:24.857690 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.857409 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb" event={"ID":"d9380eb7-166e-4602-a7c5-83b8b9e0d84f","Type":"ContainerDied","Data":"c78065f646d1e0c18e77136613b81191cf11fa666ccd3757637a4768afe6b5c7"} Oct 30 01:43:24.857690 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.857436 5055 scope.go:122] "RemoveContainer" containerID="df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e" Oct 30 01:43:24.867504 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.867477 5055 scope.go:122] "RemoveContainer" containerID="df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e" Oct 30 01:43:24.867773 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:43:24.867756 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e\": container with ID starting with df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e not found: ID does not exist" containerID="df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e" Oct 30 01:43:24.867835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.867782 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e"} err="failed to get container status \"df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e\": rpc error: code = NotFound desc = could not find container \"df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e\": container with ID starting with df2a20e5731bd72c129eb7e9980b4cb5c9d1d18a08d81dc18424dd9f9d5b918e not found: ID does not exist" Oct 30 01:43:24.909500 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.909467 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb"] Oct 30 01:43:24.921295 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:24.921272 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-6b3dbd-76f76f8d7c-s42gb"] Oct 30 01:43:26.821851 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:26.821822 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9380eb7-166e-4602-a7c5-83b8b9e0d84f" path="/var/lib/kubelet/pods/d9380eb7-166e-4602-a7c5-83b8b9e0d84f/volumes" Oct 30 01:43:43.154754 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.154720 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt"] Oct 30 01:43:43.155307 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.154957 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" podUID="857ef08b-a4e1-404b-9c40-4a0dc5744e13" containerName="maas-api" containerID="cri-o://7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c" gracePeriod=30 Oct 30 01:43:43.577905 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.577877 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:43:43.667378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.658856 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-maas-api-tls\") pod \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " Oct 30 01:43:43.667378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.659048 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-metrics-certs\") pod \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " Oct 30 01:43:43.667378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.659079 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/857ef08b-a4e1-404b-9c40-4a0dc5744e13-kube-api-access-5bcvr\" (UniqueName: \"kubernetes.io/projected/857ef08b-a4e1-404b-9c40-4a0dc5744e13-kube-api-access-5bcvr\") pod \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " Oct 30 01:43:43.667378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.659138 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/857ef08b-a4e1-404b-9c40-4a0dc5744e13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/857ef08b-a4e1-404b-9c40-4a0dc5744e13-openshift-service-ca\") pod \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\" (UID: \"857ef08b-a4e1-404b-9c40-4a0dc5744e13\") " Oct 30 01:43:43.684758 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.684700 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/857ef08b-a4e1-404b-9c40-4a0dc5744e13-openshift-service-ca" pod "857ef08b-a4e1-404b-9c40-4a0dc5744e13" (UID: "857ef08b-a4e1-404b-9c40-4a0dc5744e13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:43.768896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.760767 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/857ef08b-a4e1-404b-9c40-4a0dc5744e13-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:43:43.824457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.824365 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-maas-api-tls" pod "857ef08b-a4e1-404b-9c40-4a0dc5744e13" (UID: "857ef08b-a4e1-404b-9c40-4a0dc5744e13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:43.824860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.824840 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/857ef08b-a4e1-404b-9c40-4a0dc5744e13-kube-api-access-5bcvr" pod "857ef08b-a4e1-404b-9c40-4a0dc5744e13" (UID: "857ef08b-a4e1-404b-9c40-4a0dc5744e13"). InnerVolumeSpecName "kube-api-access-5bcvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:43.829033 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.828990 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-metrics-certs" pod "857ef08b-a4e1-404b-9c40-4a0dc5744e13" (UID: "857ef08b-a4e1-404b-9c40-4a0dc5744e13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:43.875096 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.875066 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:43:43.875403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.875386 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bcvr\" (UniqueName: \"kubernetes.io/projected/857ef08b-a4e1-404b-9c40-4a0dc5744e13-kube-api-access-5bcvr\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:43:43.875895 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:43.875878 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/857ef08b-a4e1-404b-9c40-4a0dc5744e13-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:43:44.010385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:44.000058 5055 generic.go:356] "Generic (PLEG): container finished" podID="857ef08b-a4e1-404b-9c40-4a0dc5744e13" containerID="7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c" exitCode=0 Oct 30 01:43:44.010385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:44.000171 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" event={"ID":"857ef08b-a4e1-404b-9c40-4a0dc5744e13","Type":"ContainerDied","Data":"7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c"} Oct 30 01:43:44.010385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:44.000220 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" event={"ID":"857ef08b-a4e1-404b-9c40-4a0dc5744e13","Type":"ContainerDied","Data":"287450bca25f7d6a1fa00e26ecbc65bfff342aff1eb2a8bdbf54870a7656a3ec"} Oct 30 01:43:44.010385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:44.000249 5055 scope.go:122] "RemoveContainer" containerID="7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c" Oct 30 01:43:44.010385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:44.000424 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt" Oct 30 01:43:44.041878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:44.041854 5055 scope.go:122] "RemoveContainer" containerID="7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c" Oct 30 01:43:44.042220 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:43:44.042194 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c\": container with ID starting with 7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c not found: ID does not exist" containerID="7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c" Oct 30 01:43:44.042363 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:44.042232 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c"} err="failed to get container status \"7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c\": rpc error: code = NotFound desc = could not find container \"7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c\": container with ID starting with 7122e66e735ca0189893755992c619f049d2ec734a59651ac64a9dd5cc181b9c not found: ID does not exist" Oct 30 01:43:44.067366 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:44.067156 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt"] Oct 30 01:43:44.080030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:44.079961 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-3fde1c-d64fb9df6-9zdlt"] Oct 30 01:43:44.831826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:43:44.831795 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="857ef08b-a4e1-404b-9c40-4a0dc5744e13" path="/var/lib/kubelet/pods/857ef08b-a4e1-404b-9c40-4a0dc5744e13/volumes" Oct 30 01:43:49.219702 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:43:49.219665 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:44:00.147811 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:00.147773 5055 generic.go:356] "Generic (PLEG): container finished" podID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerID="c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654" exitCode=0 Oct 30 01:44:00.148275 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:00.147842 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" event={"ID":"59c75d79-a532-4de1-9d1e-12925e33c5dd","Type":"ContainerDied","Data":"c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654"} Oct 30 01:44:00.148367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:00.148297 5055 scope.go:122] "RemoveContainer" containerID="c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654" Oct 30 01:44:00.149881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:00.149854 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:44:00.149950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:00.149923 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:44:00.149985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:00.149957 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:44:01.156753 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:01.156723 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" event={"ID":"59c75d79-a532-4de1-9d1e-12925e33c5dd","Type":"ContainerStarted","Data":"2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f"} Oct 30 01:44:01.835689 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:01.835658 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:44:01.835890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:01.835871 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:44:01.835938 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:01.835922 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 01:44:02.835495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:02.835465 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:44:02.835989 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:02.835523 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 01:44:03.835541 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:03.835494 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:44:03.835986 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:03.835560 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 01:44:03.889760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:03.887727 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p"] Oct 30 01:44:04.216525 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:04.216487 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" containerID="cri-o://c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df" gracePeriod=30 Oct 30 01:44:04.835274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:04.835247 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:44:04.835474 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:04.835299 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 01:44:05.842379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:05.836725 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Oct 30 01:44:05.842379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:05.836787 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Oct 30 01:44:06.840458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:06.840429 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:44:06.840811 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:06.840792 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:44:06.841522 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:06.841503 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:44:09.549715 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.549694 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:44:09.648179 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.648148 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-data\") pod \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " Oct 30 01:44:09.648485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.648461 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-data" pod "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" (UID: "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:09.648673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.648652 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-socket\") pod \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " Oct 30 01:44:09.648857 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.648840 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-socket" pod "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" (UID: "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:09.648857 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.648841 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-kube-api-access-ghzmd\" (UniqueName: \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-kube-api-access-ghzmd\") pod \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " Oct 30 01:44:09.649006 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.648891 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-podinfo\") pod \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " Oct 30 01:44:09.649006 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.648931 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istiod-ca-cert\") pod \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " Oct 30 01:44:09.649006 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.648974 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-credential-socket\") pod \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " Oct 30 01:44:09.649190 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.649016 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-envoy\") pod \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " Oct 30 01:44:09.649190 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.649044 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-certs\") pod \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " Oct 30 01:44:09.649190 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.649098 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-token\" (UniqueName: \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-token\") pod \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\" (UID: \"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26\") " Oct 30 01:44:09.649942 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.649491 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-credential-socket" pod "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" (UID: "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:09.649942 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.649617 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:09.649942 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.649630 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:09.649942 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.649638 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:09.649942 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.649802 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istiod-ca-cert" pod "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" (UID: "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:09.650259 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.650065 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-certs" pod "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" (UID: "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:09.651741 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.651715 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-podinfo" pod "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" (UID: "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:44:09.652483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.652457 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-envoy" pod "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" (UID: "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:09.653922 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.653890 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-token" pod "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" (UID: "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:09.654024 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.653969 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-kube-api-access-ghzmd" pod "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" (UID: "94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26"). InnerVolumeSpecName "kube-api-access-ghzmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:09.751011 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.750908 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:09.751011 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.750934 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:09.751011 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.750943 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:09.751011 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.750951 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzmd\" (UniqueName: \"kubernetes.io/projected/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-kube-api-access-ghzmd\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:09.751011 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.750985 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:09.751011 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.750993 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:09.960944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.960909 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf"] Oct 30 01:44:09.961155 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:09.961118 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" podUID="7ff14eef-46bf-4fad-88b6-06d591dfc033" containerName="maas-api" containerID="cri-o://98fe2c45206fa3904ec3c876616c09a5a0cd594f2417ab904570b6c7571f717f" gracePeriod=30 Oct 30 01:44:10.458454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.455273 5055 generic.go:356] "Generic (PLEG): container finished" podID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerID="c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df" exitCode=0 Oct 30 01:44:10.458454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.455379 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" Oct 30 01:44:10.458454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.455401 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" event={"ID":"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26","Type":"ContainerDied","Data":"c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df"} Oct 30 01:44:10.458454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.458367 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p" event={"ID":"94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26","Type":"ContainerDied","Data":"e2e334ea1b58654e25054c5c54c5ae58f619ca5bb51da19766049c6f86dafb52"} Oct 30 01:44:10.458454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.458400 5055 scope.go:122] "RemoveContainer" containerID="c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df" Oct 30 01:44:10.466015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.465989 5055 generic.go:356] "Generic (PLEG): container finished" podID="7ff14eef-46bf-4fad-88b6-06d591dfc033" containerID="98fe2c45206fa3904ec3c876616c09a5a0cd594f2417ab904570b6c7571f717f" exitCode=0 Oct 30 01:44:10.466182 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.466165 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" event={"ID":"7ff14eef-46bf-4fad-88b6-06d591dfc033","Type":"ContainerDied","Data":"98fe2c45206fa3904ec3c876616c09a5a0cd594f2417ab904570b6c7571f717f"} Oct 30 01:44:10.512001 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.510767 5055 scope.go:122] "RemoveContainer" containerID="c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df" Oct 30 01:44:10.512001 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:10.511296 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df\": container with ID starting with c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df not found: ID does not exist" containerID="c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df" Oct 30 01:44:10.512001 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.511703 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df"} err="failed to get container status \"c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df\": rpc error: code = NotFound desc = could not find container \"c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df\": container with ID starting with c1a813367528375afc222359f3cf1068831bdf4bc7bf8a3fc94aaab21c8fd5df not found: ID does not exist" Oct 30 01:44:10.623956 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.622809 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p"] Oct 30 01:44:10.648367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.647114 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-kzp7p"] Oct 30 01:44:10.677280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.677249 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:44:10.725382 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.725313 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff14eef-46bf-4fad-88b6-06d591dfc033-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff14eef-46bf-4fad-88b6-06d591dfc033-openshift-service-ca\") pod \"7ff14eef-46bf-4fad-88b6-06d591dfc033\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " Oct 30 01:44:10.725572 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.725505 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff14eef-46bf-4fad-88b6-06d591dfc033-kube-api-access-lkdwj\" (UniqueName: \"kubernetes.io/projected/7ff14eef-46bf-4fad-88b6-06d591dfc033-kube-api-access-lkdwj\") pod \"7ff14eef-46bf-4fad-88b6-06d591dfc033\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " Oct 30 01:44:10.725662 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.725624 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-maas-api-tls\") pod \"7ff14eef-46bf-4fad-88b6-06d591dfc033\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " Oct 30 01:44:10.725822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.725805 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-metrics-certs\") pod \"7ff14eef-46bf-4fad-88b6-06d591dfc033\" (UID: \"7ff14eef-46bf-4fad-88b6-06d591dfc033\") " Oct 30 01:44:10.726558 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.726515 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff14eef-46bf-4fad-88b6-06d591dfc033-openshift-service-ca" pod "7ff14eef-46bf-4fad-88b6-06d591dfc033" (UID: "7ff14eef-46bf-4fad-88b6-06d591dfc033"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:10.728504 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.728415 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-metrics-certs" pod "7ff14eef-46bf-4fad-88b6-06d591dfc033" (UID: "7ff14eef-46bf-4fad-88b6-06d591dfc033"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:10.731350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.731261 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-maas-api-tls" pod "7ff14eef-46bf-4fad-88b6-06d591dfc033" (UID: "7ff14eef-46bf-4fad-88b6-06d591dfc033"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:10.731589 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.731568 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff14eef-46bf-4fad-88b6-06d591dfc033-kube-api-access-lkdwj" pod "7ff14eef-46bf-4fad-88b6-06d591dfc033" (UID: "7ff14eef-46bf-4fad-88b6-06d591dfc033"). InnerVolumeSpecName "kube-api-access-lkdwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:10.820950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.820920 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" path="/var/lib/kubelet/pods/94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26/volumes" Oct 30 01:44:10.827391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.827368 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:10.827391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.827389 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ff14eef-46bf-4fad-88b6-06d591dfc033-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:10.827560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.827399 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkdwj\" (UniqueName: \"kubernetes.io/projected/7ff14eef-46bf-4fad-88b6-06d591dfc033-kube-api-access-lkdwj\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:10.827560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:10.827407 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ff14eef-46bf-4fad-88b6-06d591dfc033-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:11.476710 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:11.476684 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" Oct 30 01:44:11.476905 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:11.476679 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf" event={"ID":"7ff14eef-46bf-4fad-88b6-06d591dfc033","Type":"ContainerDied","Data":"d1b28392320865b03673e0a880ab56200edea1ac800eb720ff621d4a7dc149c5"} Oct 30 01:44:11.476905 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:11.476841 5055 scope.go:122] "RemoveContainer" containerID="98fe2c45206fa3904ec3c876616c09a5a0cd594f2417ab904570b6c7571f717f" Oct 30 01:44:11.529273 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:11.529244 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf"] Oct 30 01:44:11.541474 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:11.541449 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-c22e6b-7984f99fd6-b8bqf"] Oct 30 01:44:12.850342 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:12.850292 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff14eef-46bf-4fad-88b6-06d591dfc033" path="/var/lib/kubelet/pods/7ff14eef-46bf-4fad-88b6-06d591dfc033/volumes" Oct 30 01:44:14.501670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:14.501590 5055 generic.go:356] "Generic (PLEG): container finished" podID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerID="561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8" exitCode=0 Oct 30 01:44:14.502633 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:14.501661 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" event={"ID":"6ae8f195-ce05-4880-8cc0-a76799258ff0","Type":"ContainerDied","Data":"561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8"} Oct 30 01:44:14.502633 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:14.502049 5055 scope.go:122] "RemoveContainer" containerID="561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8" Oct 30 01:44:14.503550 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:14.503520 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:44:14.503672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:14.503585 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:44:14.503672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:14.503615 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:44:15.548016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:15.547980 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" event={"ID":"6ae8f195-ce05-4880-8cc0-a76799258ff0","Type":"ContainerStarted","Data":"9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e"} Oct 30 01:44:15.933116 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:15.933034 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:44:15.934206 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:15.934183 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 01:44:15.934274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:15.934236 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 01:44:16.788947 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:16.788915 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz"] Oct 30 01:44:16.789950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:16.789896 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" podUID="bfb2f44c-9d01-496e-9d63-9e79cc2c2918" containerName="maas-api" containerID="cri-o://57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9" gracePeriod=30 Oct 30 01:44:16.932646 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:16.932613 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 01:44:16.932808 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:16.932670 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 01:44:17.451349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.451238 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:44:17.530404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.528278 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-kube-api-access-ndtdf\" (UniqueName: \"kubernetes.io/projected/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-kube-api-access-ndtdf\") pod \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " Oct 30 01:44:17.530404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.528346 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-openshift-service-ca\") pod \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " Oct 30 01:44:17.530404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.528375 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-metrics-certs\") pod \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " Oct 30 01:44:17.530404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.528417 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-maas-api-tls\") pod \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\" (UID: \"bfb2f44c-9d01-496e-9d63-9e79cc2c2918\") " Oct 30 01:44:17.530404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.528816 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-openshift-service-ca" pod "bfb2f44c-9d01-496e-9d63-9e79cc2c2918" (UID: "bfb2f44c-9d01-496e-9d63-9e79cc2c2918"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:17.530404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.529167 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:17.531730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.531706 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-maas-api-tls" pod "bfb2f44c-9d01-496e-9d63-9e79cc2c2918" (UID: "bfb2f44c-9d01-496e-9d63-9e79cc2c2918"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:17.533014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.532976 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-metrics-certs" pod "bfb2f44c-9d01-496e-9d63-9e79cc2c2918" (UID: "bfb2f44c-9d01-496e-9d63-9e79cc2c2918"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:17.533176 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.533155 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-kube-api-access-ndtdf" pod "bfb2f44c-9d01-496e-9d63-9e79cc2c2918" (UID: "bfb2f44c-9d01-496e-9d63-9e79cc2c2918"). InnerVolumeSpecName "kube-api-access-ndtdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:17.609728 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.608084 5055 generic.go:356] "Generic (PLEG): container finished" podID="bfb2f44c-9d01-496e-9d63-9e79cc2c2918" containerID="57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9" exitCode=0 Oct 30 01:44:17.609728 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.608275 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" event={"ID":"bfb2f44c-9d01-496e-9d63-9e79cc2c2918","Type":"ContainerDied","Data":"57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9"} Oct 30 01:44:17.609728 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.608365 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" event={"ID":"bfb2f44c-9d01-496e-9d63-9e79cc2c2918","Type":"ContainerDied","Data":"ff1fd84c7bd4715f5fb4defe3a55d52d1daa16be118d7a96bb329345f6edc3b4"} Oct 30 01:44:17.609728 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.608399 5055 scope.go:122] "RemoveContainer" containerID="57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9" Oct 30 01:44:17.609728 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.608672 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz" Oct 30 01:44:17.629500 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.629482 5055 scope.go:122] "RemoveContainer" containerID="57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9" Oct 30 01:44:17.629814 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:17.629792 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9\": container with ID starting with 57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9 not found: ID does not exist" containerID="57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9" Oct 30 01:44:17.629899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.629826 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9"} err="failed to get container status \"57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9\": rpc error: code = NotFound desc = could not find container \"57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9\": container with ID starting with 57931b1068b70f2aca0035f71ec68e71665cd7f8e99af2e5aadb47778b9654e9 not found: ID does not exist" Oct 30 01:44:17.630391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.630371 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndtdf\" (UniqueName: \"kubernetes.io/projected/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-kube-api-access-ndtdf\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:17.630463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.630413 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:17.630463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.630423 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfb2f44c-9d01-496e-9d63-9e79cc2c2918-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:17.673574 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.673534 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz"] Oct 30 01:44:17.689351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.689083 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-c9d174-7f9469cc8f-qlfwz"] Oct 30 01:44:17.932849 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.932764 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 01:44:17.932849 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:17.932814 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 01:44:18.821631 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:18.821599 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfb2f44c-9d01-496e-9d63-9e79cc2c2918" path="/var/lib/kubelet/pods/bfb2f44c-9d01-496e-9d63-9e79cc2c2918/volumes" Oct 30 01:44:18.822002 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:18.821988 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:44:18.932851 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:18.932814 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 01:44:18.933008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:18.932875 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 01:44:19.218665 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:19.218627 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:44:19.842492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:19.842196 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7"] Oct 30 01:44:19.842492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:19.842475 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" containerID="cri-o://1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1" gracePeriod=30 Oct 30 01:44:19.932837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:19.932807 5055 patch_prober.go:29] interesting pod/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 30 01:44:19.933038 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:19.932875 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 30 01:44:20.938730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:20.938701 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:44:20.939865 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:20.939842 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:44:25.258170 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.258145 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:44:25.366711 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.366680 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-credential-socket\") pod \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " Oct 30 01:44:25.367349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.367311 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-token\" (UniqueName: \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-token\") pod \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " Oct 30 01:44:25.367870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.367852 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-envoy\") pod \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " Oct 30 01:44:25.369202 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.369182 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-kube-api-access-lgxdp\" (UniqueName: \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-kube-api-access-lgxdp\") pod \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " Oct 30 01:44:25.369812 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.369793 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-certs\") pod \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " Oct 30 01:44:25.370022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.370003 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-socket\") pod \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " Oct 30 01:44:25.370282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.370265 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-podinfo\") pod \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " Oct 30 01:44:25.370465 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.367135 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-credential-socket" pod "1bafa687-8080-4ee5-9388-fb0d6d74ffc1" (UID: "1bafa687-8080-4ee5-9388-fb0d6d74ffc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:25.370593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.370103 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-certs" pod "1bafa687-8080-4ee5-9388-fb0d6d74ffc1" (UID: "1bafa687-8080-4ee5-9388-fb0d6d74ffc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:25.370902 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.370282 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-socket" pod "1bafa687-8080-4ee5-9388-fb0d6d74ffc1" (UID: "1bafa687-8080-4ee5-9388-fb0d6d74ffc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:25.371084 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.371069 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istiod-ca-cert\") pod \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " Oct 30 01:44:25.372591 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.372563 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-data\") pod \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\" (UID: \"1bafa687-8080-4ee5-9388-fb0d6d74ffc1\") " Oct 30 01:44:25.373921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.373899 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.374090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.374076 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.374216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.374204 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.374314 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.371489 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istiod-ca-cert" pod "1bafa687-8080-4ee5-9388-fb0d6d74ffc1" (UID: "1bafa687-8080-4ee5-9388-fb0d6d74ffc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:25.374489 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.372981 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-data" pod "1bafa687-8080-4ee5-9388-fb0d6d74ffc1" (UID: "1bafa687-8080-4ee5-9388-fb0d6d74ffc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:25.378702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.378664 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-envoy" pod "1bafa687-8080-4ee5-9388-fb0d6d74ffc1" (UID: "1bafa687-8080-4ee5-9388-fb0d6d74ffc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:25.378971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.378923 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-token" pod "1bafa687-8080-4ee5-9388-fb0d6d74ffc1" (UID: "1bafa687-8080-4ee5-9388-fb0d6d74ffc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:25.381010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.380983 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-podinfo" pod "1bafa687-8080-4ee5-9388-fb0d6d74ffc1" (UID: "1bafa687-8080-4ee5-9388-fb0d6d74ffc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:44:25.381994 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.381975 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-kube-api-access-lgxdp" pod "1bafa687-8080-4ee5-9388-fb0d6d74ffc1" (UID: "1bafa687-8080-4ee5-9388-fb0d6d74ffc1"). InnerVolumeSpecName "kube-api-access-lgxdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:25.475907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.475877 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.475907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.475901 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.475907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.475910 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgxdp\" (UniqueName: \"kubernetes.io/projected/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-kube-api-access-lgxdp\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.476120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.475919 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.476120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.475927 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.476120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.475935 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1bafa687-8080-4ee5-9388-fb0d6d74ffc1-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:25.668771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.668405 5055 generic.go:356] "Generic (PLEG): container finished" podID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerID="1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1" exitCode=0 Oct 30 01:44:25.668771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.668473 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" Oct 30 01:44:25.668771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.668510 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" event={"ID":"1bafa687-8080-4ee5-9388-fb0d6d74ffc1","Type":"ContainerDied","Data":"1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1"} Oct 30 01:44:25.668771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.668559 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7" event={"ID":"1bafa687-8080-4ee5-9388-fb0d6d74ffc1","Type":"ContainerDied","Data":"2a3e2af55de785494886f7c12ede25ab7bf09f534fef7e669fcb6acb442c28bc"} Oct 30 01:44:25.668771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.668586 5055 scope.go:122] "RemoveContainer" containerID="1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1" Oct 30 01:44:25.682134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.682116 5055 scope.go:122] "RemoveContainer" containerID="9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad" Oct 30 01:44:25.691674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.691659 5055 scope.go:122] "RemoveContainer" containerID="1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1" Oct 30 01:44:25.691955 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:25.691934 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1\": container with ID starting with 1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1 not found: ID does not exist" containerID="1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1" Oct 30 01:44:25.692044 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.691962 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1"} err="failed to get container status \"1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1\": rpc error: code = NotFound desc = could not find container \"1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1\": container with ID starting with 1fac3524bf9e56739da43fa50e0d126719c5799f3dac33c78b08cf3278b073e1 not found: ID does not exist" Oct 30 01:44:25.692044 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.691978 5055 scope.go:122] "RemoveContainer" containerID="9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad" Oct 30 01:44:25.692253 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:25.692235 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad\": container with ID starting with 9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad not found: ID does not exist" containerID="9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad" Oct 30 01:44:25.692363 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.692263 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad"} err="failed to get container status \"9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad\": rpc error: code = NotFound desc = could not find container \"9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad\": container with ID starting with 9cd5c2869e9f433f0a2725e1efb8b4573c38668193fca8857174703b919cbbad not found: ID does not exist" Oct 30 01:44:25.726001 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.725977 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7"] Oct 30 01:44:25.738261 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:25.738234 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-c22e6b-openshift-default-bf5597b77-9pqc7"] Oct 30 01:44:26.822032 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:26.822001 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" path="/var/lib/kubelet/pods/1bafa687-8080-4ee5-9388-fb0d6d74ffc1/volumes" Oct 30 01:44:29.927235 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:29.927204 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn"] Oct 30 01:44:29.927688 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:29.927468 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" podUID="673cf0f4-cf51-4653-bc59-c79408b148ba" containerName="maas-api" containerID="cri-o://8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964" gracePeriod=30 Oct 30 01:44:30.269650 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.269624 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:44:30.381250 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.381224 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673cf0f4-cf51-4653-bc59-c79408b148ba-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673cf0f4-cf51-4653-bc59-c79408b148ba-openshift-service-ca\") pod \"673cf0f4-cf51-4653-bc59-c79408b148ba\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " Oct 30 01:44:30.381431 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.381260 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-maas-api-tls\") pod \"673cf0f4-cf51-4653-bc59-c79408b148ba\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " Oct 30 01:44:30.381431 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.381279 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-metrics-certs\" (UniqueName: \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-metrics-certs\") pod \"673cf0f4-cf51-4653-bc59-c79408b148ba\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " Oct 30 01:44:30.381574 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.381463 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673cf0f4-cf51-4653-bc59-c79408b148ba-kube-api-access-ccfjr\" (UniqueName: \"kubernetes.io/projected/673cf0f4-cf51-4653-bc59-c79408b148ba-kube-api-access-ccfjr\") pod \"673cf0f4-cf51-4653-bc59-c79408b148ba\" (UID: \"673cf0f4-cf51-4653-bc59-c79408b148ba\") " Oct 30 01:44:30.382124 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.382095 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673cf0f4-cf51-4653-bc59-c79408b148ba-openshift-service-ca" pod "673cf0f4-cf51-4653-bc59-c79408b148ba" (UID: "673cf0f4-cf51-4653-bc59-c79408b148ba"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:30.384371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.384297 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-maas-api-tls" pod "673cf0f4-cf51-4653-bc59-c79408b148ba" (UID: "673cf0f4-cf51-4653-bc59-c79408b148ba"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:30.384371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.384359 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673cf0f4-cf51-4653-bc59-c79408b148ba-kube-api-access-ccfjr" pod "673cf0f4-cf51-4653-bc59-c79408b148ba" (UID: "673cf0f4-cf51-4653-bc59-c79408b148ba"). InnerVolumeSpecName "kube-api-access-ccfjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:30.384513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.384397 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-metrics-certs" pod "673cf0f4-cf51-4653-bc59-c79408b148ba" (UID: "673cf0f4-cf51-4653-bc59-c79408b148ba"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:30.488405 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.487319 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/673cf0f4-cf51-4653-bc59-c79408b148ba-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.488405 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.487378 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.488405 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.487394 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/673cf0f4-cf51-4653-bc59-c79408b148ba-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.488405 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.487407 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccfjr\" (UniqueName: \"kubernetes.io/projected/673cf0f4-cf51-4653-bc59-c79408b148ba-kube-api-access-ccfjr\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:30.713371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.713303 5055 generic.go:356] "Generic (PLEG): container finished" podID="673cf0f4-cf51-4653-bc59-c79408b148ba" containerID="8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964" exitCode=0 Oct 30 01:44:30.713530 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.713387 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" event={"ID":"673cf0f4-cf51-4653-bc59-c79408b148ba","Type":"ContainerDied","Data":"8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964"} Oct 30 01:44:30.713530 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.713450 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" event={"ID":"673cf0f4-cf51-4653-bc59-c79408b148ba","Type":"ContainerDied","Data":"569dcd79c81e039bb55c3789d22a930323ae06321454db93e80fd71275e2f897"} Oct 30 01:44:30.713530 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.713402 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn" Oct 30 01:44:30.713690 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.713478 5055 scope.go:122] "RemoveContainer" containerID="8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964" Oct 30 01:44:30.731247 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.730818 5055 scope.go:122] "RemoveContainer" containerID="8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964" Oct 30 01:44:30.731565 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:30.731544 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964\": container with ID starting with 8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964 not found: ID does not exist" containerID="8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964" Oct 30 01:44:30.731668 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.731578 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964"} err="failed to get container status \"8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964\": rpc error: code = NotFound desc = could not find container \"8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964\": container with ID starting with 8e7fff9921c2dd7fd9b52cd80c3bfafaddac34ee21f5940e27d7152ef2baa964 not found: ID does not exist" Oct 30 01:44:30.796473 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.796314 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn"] Oct 30 01:44:30.826042 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:30.826011 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ea0fc4-6f54c7b576-jd5vn"] Oct 30 01:44:32.821916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:32.821883 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673cf0f4-cf51-4653-bc59-c79408b148ba" path="/var/lib/kubelet/pods/673cf0f4-cf51-4653-bc59-c79408b148ba/volumes" Oct 30 01:44:39.342175 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:39.342144 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd"] Oct 30 01:44:39.342679 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:39.342374 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" containerID="cri-o://0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6" gracePeriod=30 Oct 30 01:44:39.756240 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:39.756206 5055 patch_prober.go:29] interesting pod/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:44:39.756441 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:39.756265 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:44:44.326170 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.326133 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb"] Oct 30 01:44:44.333866 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.326433 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" containerID="cri-o://2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f" gracePeriod=30 Oct 30 01:44:44.651838 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.651810 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:44:44.705073 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705046 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-envoy\") pod \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " Oct 30 01:44:44.705226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705123 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-token\" (UniqueName: \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-token\") pod \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " Oct 30 01:44:44.705226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705144 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istiod-ca-cert\") pod \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " Oct 30 01:44:44.705226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705171 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-kube-api-access-46nns\" (UniqueName: \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-kube-api-access-46nns\") pod \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " Oct 30 01:44:44.705226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705194 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-credential-socket\") pod \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " Oct 30 01:44:44.705490 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705281 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-certs\") pod \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " Oct 30 01:44:44.705490 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705319 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-podinfo\") pod \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " Oct 30 01:44:44.705490 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705404 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-data\") pod \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " Oct 30 01:44:44.705490 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705453 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-socket\") pod \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\" (UID: \"c94cdcb7-32d4-437f-af1f-df2a30c79d3b\") " Oct 30 01:44:44.705730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705707 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-certs" pod "c94cdcb7-32d4-437f-af1f-df2a30c79d3b" (UID: "c94cdcb7-32d4-437f-af1f-df2a30c79d3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:44.705800 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705724 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-credential-socket" pod "c94cdcb7-32d4-437f-af1f-df2a30c79d3b" (UID: "c94cdcb7-32d4-437f-af1f-df2a30c79d3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:44.705973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705807 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-data" pod "c94cdcb7-32d4-437f-af1f-df2a30c79d3b" (UID: "c94cdcb7-32d4-437f-af1f-df2a30c79d3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:44.705973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705901 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-socket" pod "c94cdcb7-32d4-437f-af1f-df2a30c79d3b" (UID: "c94cdcb7-32d4-437f-af1f-df2a30c79d3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:44.705973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.705975 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istiod-ca-cert" pod "c94cdcb7-32d4-437f-af1f-df2a30c79d3b" (UID: "c94cdcb7-32d4-437f-af1f-df2a30c79d3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:44.706222 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.706191 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:44.706222 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.706216 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:44.706388 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.706231 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:44.706388 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.706244 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:44.706388 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.706258 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:44.707641 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.707619 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-envoy" pod "c94cdcb7-32d4-437f-af1f-df2a30c79d3b" (UID: "c94cdcb7-32d4-437f-af1f-df2a30c79d3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:44.707860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.707837 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-kube-api-access-46nns" pod "c94cdcb7-32d4-437f-af1f-df2a30c79d3b" (UID: "c94cdcb7-32d4-437f-af1f-df2a30c79d3b"). InnerVolumeSpecName "kube-api-access-46nns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:44.707860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.707845 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-token" pod "c94cdcb7-32d4-437f-af1f-df2a30c79d3b" (UID: "c94cdcb7-32d4-437f-af1f-df2a30c79d3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:44.707960 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.707942 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-podinfo" pod "c94cdcb7-32d4-437f-af1f-df2a30c79d3b" (UID: "c94cdcb7-32d4-437f-af1f-df2a30c79d3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:44:44.807641 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.807613 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:44.807641 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.807637 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:44.807839 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.807649 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46nns\" (UniqueName: \"kubernetes.io/projected/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-kube-api-access-46nns\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:44.807839 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.807661 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c94cdcb7-32d4-437f-af1f-df2a30c79d3b-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:44.814531 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.814503 5055 generic.go:356] "Generic (PLEG): container finished" podID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerID="0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6" exitCode=0 Oct 30 01:44:44.814663 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.814630 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" Oct 30 01:44:44.820764 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.820733 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" event={"ID":"c94cdcb7-32d4-437f-af1f-df2a30c79d3b","Type":"ContainerDied","Data":"0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6"} Oct 30 01:44:44.820883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.820770 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd" event={"ID":"c94cdcb7-32d4-437f-af1f-df2a30c79d3b","Type":"ContainerDied","Data":"13bdfcd3304d8892ce839ce3ada8a5922b807babc037f7d8e6e8e317ed0f1a7e"} Oct 30 01:44:44.820883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.820800 5055 scope.go:122] "RemoveContainer" containerID="0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6" Oct 30 01:44:44.833246 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.833232 5055 scope.go:122] "RemoveContainer" containerID="1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39" Oct 30 01:44:44.842061 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.842043 5055 scope.go:122] "RemoveContainer" containerID="0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6" Oct 30 01:44:44.842319 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:44.842301 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6\": container with ID starting with 0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6 not found: ID does not exist" containerID="0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6" Oct 30 01:44:44.842404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.842349 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6"} err="failed to get container status \"0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6\": rpc error: code = NotFound desc = could not find container \"0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6\": container with ID starting with 0cc41d2fa25868646fff8eb476cad2448865e7f2e77f08e9d182d584949019e6 not found: ID does not exist" Oct 30 01:44:44.842404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.842371 5055 scope.go:122] "RemoveContainer" containerID="1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39" Oct 30 01:44:44.842627 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:44.842610 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39\": container with ID starting with 1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39 not found: ID does not exist" containerID="1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39" Oct 30 01:44:44.842678 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.842632 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39"} err="failed to get container status \"1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39\": rpc error: code = NotFound desc = could not find container \"1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39\": container with ID starting with 1805971ab7eaa9e0f14de52f6d4e11425b90c8646bf8eba2931ec779e3a36d39 not found: ID does not exist" Oct 30 01:44:44.872123 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.872099 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd"] Oct 30 01:44:44.876222 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:44.876165 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ea0fc4-openshift-default-5544d5887-hwjdd"] Oct 30 01:44:46.821883 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:46.821851 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" path="/var/lib/kubelet/pods/c94cdcb7-32d4-437f-af1f-df2a30c79d3b/volumes" Oct 30 01:44:49.219092 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:49.219062 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:44:49.602505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.602475 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:44:49.680809 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.680783 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c75d79-a532-4de1-9d1e-12925e33c5dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c75d79-a532-4de1-9d1e-12925e33c5dd-istiod-ca-cert\") pod \"59c75d79-a532-4de1-9d1e-12925e33c5dd\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " Oct 30 01:44:49.680972 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.680840 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-socket\") pod \"59c75d79-a532-4de1-9d1e-12925e33c5dd\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " Oct 30 01:44:49.680972 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.680876 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-data\") pod \"59c75d79-a532-4de1-9d1e-12925e33c5dd\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " Oct 30 01:44:49.681088 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.681028 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-certs\") pod \"59c75d79-a532-4de1-9d1e-12925e33c5dd\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " Oct 30 01:44:49.681141 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.681123 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-envoy\") pod \"59c75d79-a532-4de1-9d1e-12925e33c5dd\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " Oct 30 01:44:49.681194 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.681159 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-token\" (UniqueName: \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-token\") pod \"59c75d79-a532-4de1-9d1e-12925e33c5dd\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " Oct 30 01:44:49.681251 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.681193 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-podinfo\") pod \"59c75d79-a532-4de1-9d1e-12925e33c5dd\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " Oct 30 01:44:49.681251 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.681234 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-kube-api-access-2bfzq\" (UniqueName: \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-kube-api-access-2bfzq\") pod \"59c75d79-a532-4de1-9d1e-12925e33c5dd\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " Oct 30 01:44:49.681399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.681270 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-credential-socket\") pod \"59c75d79-a532-4de1-9d1e-12925e33c5dd\" (UID: \"59c75d79-a532-4de1-9d1e-12925e33c5dd\") " Oct 30 01:44:49.681399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.681285 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-data" pod "59c75d79-a532-4de1-9d1e-12925e33c5dd" (UID: "59c75d79-a532-4de1-9d1e-12925e33c5dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:49.681706 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.681683 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-socket" pod "59c75d79-a532-4de1-9d1e-12925e33c5dd" (UID: "59c75d79-a532-4de1-9d1e-12925e33c5dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:49.681845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.681827 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:49.681916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.681855 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:49.682184 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.682157 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-certs" pod "59c75d79-a532-4de1-9d1e-12925e33c5dd" (UID: "59c75d79-a532-4de1-9d1e-12925e33c5dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:49.682377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.682356 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-credential-socket" pod "59c75d79-a532-4de1-9d1e-12925e33c5dd" (UID: "59c75d79-a532-4de1-9d1e-12925e33c5dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:49.682627 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.682608 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c75d79-a532-4de1-9d1e-12925e33c5dd-istiod-ca-cert" pod "59c75d79-a532-4de1-9d1e-12925e33c5dd" (UID: "59c75d79-a532-4de1-9d1e-12925e33c5dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:49.684780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.684754 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-token" pod "59c75d79-a532-4de1-9d1e-12925e33c5dd" (UID: "59c75d79-a532-4de1-9d1e-12925e33c5dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:49.685063 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.685043 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-podinfo" pod "59c75d79-a532-4de1-9d1e-12925e33c5dd" (UID: "59c75d79-a532-4de1-9d1e-12925e33c5dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:44:49.685837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.685817 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-kube-api-access-2bfzq" pod "59c75d79-a532-4de1-9d1e-12925e33c5dd" (UID: "59c75d79-a532-4de1-9d1e-12925e33c5dd"). InnerVolumeSpecName "kube-api-access-2bfzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:49.693359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.687677 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-envoy" pod "59c75d79-a532-4de1-9d1e-12925e33c5dd" (UID: "59c75d79-a532-4de1-9d1e-12925e33c5dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:44:49.783636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.783568 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c75d79-a532-4de1-9d1e-12925e33c5dd-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:49.783636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.783599 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:49.783636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.783612 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:49.783636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.783623 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:49.783636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.783635 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c75d79-a532-4de1-9d1e-12925e33c5dd-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:49.783885 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.783648 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bfzq\" (UniqueName: \"kubernetes.io/projected/59c75d79-a532-4de1-9d1e-12925e33c5dd-kube-api-access-2bfzq\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:49.783885 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.783661 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c75d79-a532-4de1-9d1e-12925e33c5dd-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:44:49.875476 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.875445 5055 generic.go:356] "Generic (PLEG): container finished" podID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerID="2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f" exitCode=0 Oct 30 01:44:49.875637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.875500 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" Oct 30 01:44:49.875637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.875528 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" event={"ID":"59c75d79-a532-4de1-9d1e-12925e33c5dd","Type":"ContainerDied","Data":"2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f"} Oct 30 01:44:49.875637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.875583 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb" event={"ID":"59c75d79-a532-4de1-9d1e-12925e33c5dd","Type":"ContainerDied","Data":"cdf91b85038c0c7a3f74d9906d6928071d15ab7727eef100600e5cbfacd20318"} Oct 30 01:44:49.875637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.875605 5055 scope.go:122] "RemoveContainer" containerID="2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f" Oct 30 01:44:49.886114 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.885985 5055 scope.go:122] "RemoveContainer" containerID="c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654" Oct 30 01:44:49.895515 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.895499 5055 scope.go:122] "RemoveContainer" containerID="2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f" Oct 30 01:44:49.895802 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:49.895779 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f\": container with ID starting with 2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f not found: ID does not exist" containerID="2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f" Oct 30 01:44:49.895915 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.895805 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f"} err="failed to get container status \"2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f\": rpc error: code = NotFound desc = could not find container \"2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f\": container with ID starting with 2ca5edbd19b4e185f23418b7138ea3efd57a7e1e04e1778693d2ebea28a6190f not found: ID does not exist" Oct 30 01:44:49.895915 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.895822 5055 scope.go:122] "RemoveContainer" containerID="c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654" Oct 30 01:44:49.896051 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:44:49.896029 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654\": container with ID starting with c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654 not found: ID does not exist" containerID="c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654" Oct 30 01:44:49.896112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.896062 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654"} err="failed to get container status \"c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654\": rpc error: code = NotFound desc = could not find container \"c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654\": container with ID starting with c621dc0679f868afcb544f63ff92bd27e278db9b6067a99ca1df2255f0402654 not found: ID does not exist" Oct 30 01:44:49.939022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.939000 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb"] Oct 30 01:44:49.945924 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:49.945905 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-be8a7d-openshift-default-6bb97f477c-mvskb"] Oct 30 01:44:50.819835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:44:50.819807 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" path="/var/lib/kubelet/pods/59c75d79-a532-4de1-9d1e-12925e33c5dd/volumes" Oct 30 01:45:00.177045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.176955 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs"] Oct 30 01:45:00.177581 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177564 5055 cpu_manager.go:414] "containerMap: removing container" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" Oct 30 01:45:00.177648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177582 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" Oct 30 01:45:00.177648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177598 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7ff14eef-46bf-4fad-88b6-06d591dfc033" containerName="maas-api" Oct 30 01:45:00.177648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177603 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff14eef-46bf-4fad-88b6-06d591dfc033" containerName="maas-api" Oct 30 01:45:00.177648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177618 5055 cpu_manager.go:414] "containerMap: removing container" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" Oct 30 01:45:00.177648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177623 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" Oct 30 01:45:00.177648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177636 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d9380eb7-166e-4602-a7c5-83b8b9e0d84f" containerName="maas-api" Oct 30 01:45:00.177648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177642 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9380eb7-166e-4602-a7c5-83b8b9e0d84f" containerName="maas-api" Oct 30 01:45:00.177648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177651 5055 cpu_manager.go:414] "containerMap: removing container" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177655 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177667 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177675 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177686 5055 cpu_manager.go:414] "containerMap: removing container" podUID="bfb2f44c-9d01-496e-9d63-9e79cc2c2918" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177690 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfb2f44c-9d01-496e-9d63-9e79cc2c2918" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177704 5055 cpu_manager.go:414] "containerMap: removing container" podUID="857ef08b-a4e1-404b-9c40-4a0dc5744e13" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177709 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="857ef08b-a4e1-404b-9c40-4a0dc5744e13" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177715 5055 cpu_manager.go:414] "containerMap: removing container" podUID="673cf0f4-cf51-4653-bc59-c79408b148ba" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177720 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673cf0f4-cf51-4653-bc59-c79408b148ba" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177727 5055 cpu_manager.go:414] "containerMap: removing container" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177731 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177737 5055 cpu_manager.go:414] "containerMap: removing container" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177743 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177856 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff14eef-46bf-4fad-88b6-06d591dfc033" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177867 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177874 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9380eb7-166e-4602-a7c5-83b8b9e0d84f" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177880 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c75d79-a532-4de1-9d1e-12925e33c5dd" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177890 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177899 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfb2f44c-9d01-496e-9d63-9e79cc2c2918" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177910 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="857ef08b-a4e1-404b-9c40-4a0dc5744e13" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177918 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="673cf0f4-cf51-4653-bc59-c79408b148ba" containerName="maas-api" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177926 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177933 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f6b6c6-f9ec-4bcf-abf3-e3498f20ba26" containerName="istio-proxy" Oct 30 01:45:00.178030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.177941 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" Oct 30 01:45:00.183103 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.183076 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" Oct 30 01:45:00.184364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.184342 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9"] Oct 30 01:45:00.184789 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.184776 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" Oct 30 01:45:00.184789 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.184789 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94cdcb7-32d4-437f-af1f-df2a30c79d3b" containerName="istio-proxy" Oct 30 01:45:00.184923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.184871 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bafa687-8080-4ee5-9388-fb0d6d74ffc1" containerName="istio-proxy" Oct 30 01:45:00.187886 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.187867 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" Oct 30 01:45:00.208496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.208473 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:45:00.217546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.217518 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94"] Oct 30 01:45:00.221819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.221801 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs"] Oct 30 01:45:00.221931 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.221873 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" Oct 30 01:45:00.222660 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.222639 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk"] Oct 30 01:45:00.226956 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.226935 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" Oct 30 01:45:00.231788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.231763 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9"] Oct 30 01:45:00.249359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.249300 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94"] Oct 30 01:45:00.250105 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.250081 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f54c97f-7992-47fd-8695-9405fbceb7a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9\" (UID: \"4f54c97f-7992-47fd-8695-9405fbceb7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" Oct 30 01:45:00.250386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.250362 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgds6\" (UniqueName: \"kubernetes.io/projected/4f54c97f-7992-47fd-8695-9405fbceb7a7-kube-api-access-zgds6\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9\" (UID: \"4f54c97f-7992-47fd-8695-9405fbceb7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" Oct 30 01:45:00.250504 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.250429 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mln9m\" (UniqueName: \"kubernetes.io/projected/7e963382-3d1d-4cd1-8b63-ecab9d057801-kube-api-access-mln9m\") pod \"maas-api-key-cleanup-29888745-6l2cs\" (UID: \"7e963382-3d1d-4cd1-8b63-ecab9d057801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" Oct 30 01:45:00.250591 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.250576 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e963382-3d1d-4cd1-8b63-ecab9d057801-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888745-6l2cs\" (UID: \"7e963382-3d1d-4cd1-8b63-ecab9d057801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" Oct 30 01:45:00.261443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.261410 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk"] Oct 30 01:45:00.353348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.352398 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffae23ed-c896-4b25-89cd-759b22580968-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk\" (UID: \"ffae23ed-c896-4b25-89cd-759b22580968\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" Oct 30 01:45:00.353348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.352445 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlv9w\" (UniqueName: \"kubernetes.io/projected/9a0d887c-1b6b-48f8-8f24-4e1f31599281-kube-api-access-rlv9w\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94\" (UID: \"9a0d887c-1b6b-48f8-8f24-4e1f31599281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" Oct 30 01:45:00.353348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.352494 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mln9m\" (UniqueName: \"kubernetes.io/projected/7e963382-3d1d-4cd1-8b63-ecab9d057801-kube-api-access-mln9m\") pod \"maas-api-key-cleanup-29888745-6l2cs\" (UID: \"7e963382-3d1d-4cd1-8b63-ecab9d057801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" Oct 30 01:45:00.353348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.352587 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h5jv\" (UniqueName: \"kubernetes.io/projected/ffae23ed-c896-4b25-89cd-759b22580968-kube-api-access-5h5jv\") pod \"maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk\" (UID: \"ffae23ed-c896-4b25-89cd-759b22580968\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" Oct 30 01:45:00.353348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.352668 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a0d887c-1b6b-48f8-8f24-4e1f31599281-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94\" (UID: \"9a0d887c-1b6b-48f8-8f24-4e1f31599281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" Oct 30 01:45:00.353348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.352701 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e963382-3d1d-4cd1-8b63-ecab9d057801-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888745-6l2cs\" (UID: \"7e963382-3d1d-4cd1-8b63-ecab9d057801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" Oct 30 01:45:00.353348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.352806 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f54c97f-7992-47fd-8695-9405fbceb7a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9\" (UID: \"4f54c97f-7992-47fd-8695-9405fbceb7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" Oct 30 01:45:00.353348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.352863 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgds6\" (UniqueName: \"kubernetes.io/projected/4f54c97f-7992-47fd-8695-9405fbceb7a7-kube-api-access-zgds6\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9\" (UID: \"4f54c97f-7992-47fd-8695-9405fbceb7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" Oct 30 01:45:00.359349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.354109 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e963382-3d1d-4cd1-8b63-ecab9d057801-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888745-6l2cs\" (UID: \"7e963382-3d1d-4cd1-8b63-ecab9d057801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" Oct 30 01:45:00.359349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.354742 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f54c97f-7992-47fd-8695-9405fbceb7a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9\" (UID: \"4f54c97f-7992-47fd-8695-9405fbceb7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" Oct 30 01:45:00.384594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.384528 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mln9m\" (UniqueName: \"kubernetes.io/projected/7e963382-3d1d-4cd1-8b63-ecab9d057801-kube-api-access-mln9m\") pod \"maas-api-key-cleanup-29888745-6l2cs\" (UID: \"7e963382-3d1d-4cd1-8b63-ecab9d057801\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" Oct 30 01:45:00.387907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.387856 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgds6\" (UniqueName: \"kubernetes.io/projected/4f54c97f-7992-47fd-8695-9405fbceb7a7-kube-api-access-zgds6\") pod \"maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9\" (UID: \"4f54c97f-7992-47fd-8695-9405fbceb7a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" Oct 30 01:45:00.454072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.454047 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h5jv\" (UniqueName: \"kubernetes.io/projected/ffae23ed-c896-4b25-89cd-759b22580968-kube-api-access-5h5jv\") pod \"maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk\" (UID: \"ffae23ed-c896-4b25-89cd-759b22580968\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" Oct 30 01:45:00.454210 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.454110 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a0d887c-1b6b-48f8-8f24-4e1f31599281-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94\" (UID: \"9a0d887c-1b6b-48f8-8f24-4e1f31599281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" Oct 30 01:45:00.454210 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.454205 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffae23ed-c896-4b25-89cd-759b22580968-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk\" (UID: \"ffae23ed-c896-4b25-89cd-759b22580968\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" Oct 30 01:45:00.454383 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.454228 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlv9w\" (UniqueName: \"kubernetes.io/projected/9a0d887c-1b6b-48f8-8f24-4e1f31599281-kube-api-access-rlv9w\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94\" (UID: \"9a0d887c-1b6b-48f8-8f24-4e1f31599281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" Oct 30 01:45:00.454919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.454899 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffae23ed-c896-4b25-89cd-759b22580968-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk\" (UID: \"ffae23ed-c896-4b25-89cd-759b22580968\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" Oct 30 01:45:00.454971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.454899 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a0d887c-1b6b-48f8-8f24-4e1f31599281-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94\" (UID: \"9a0d887c-1b6b-48f8-8f24-4e1f31599281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" Oct 30 01:45:00.476796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.476776 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h5jv\" (UniqueName: \"kubernetes.io/projected/ffae23ed-c896-4b25-89cd-759b22580968-kube-api-access-5h5jv\") pod \"maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk\" (UID: \"ffae23ed-c896-4b25-89cd-759b22580968\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" Oct 30 01:45:00.483717 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.483699 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlv9w\" (UniqueName: \"kubernetes.io/projected/9a0d887c-1b6b-48f8-8f24-4e1f31599281-kube-api-access-rlv9w\") pod \"maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94\" (UID: \"9a0d887c-1b6b-48f8-8f24-4e1f31599281\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" Oct 30 01:45:00.510388 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.501611 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" Oct 30 01:45:00.526296 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.526073 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" Oct 30 01:45:00.542850 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.542547 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" Oct 30 01:45:00.566392 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.566368 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" Oct 30 01:45:00.880553 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:45:00.880520 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0d887c_1b6b_48f8_8f24_4e1f31599281.slice/crio-00281a8de008af926ddffd50f548e4dbddcf45084f60dbf48b2bb57942496595 WatchSource:0}: Error finding container 00281a8de008af926ddffd50f548e4dbddcf45084f60dbf48b2bb57942496595: Status 404 returned error can't find the container with id 00281a8de008af926ddffd50f548e4dbddcf45084f60dbf48b2bb57942496595 Oct 30 01:45:00.882424 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.882401 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94"] Oct 30 01:45:00.894401 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:45:00.893588 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e963382_3d1d_4cd1_8b63_ecab9d057801.slice/crio-5fcf880eb94426993d275bd8a98c8aa4669c7140aeb00f6a609030671b6fd644 WatchSource:0}: Error finding container 5fcf880eb94426993d275bd8a98c8aa4669c7140aeb00f6a609030671b6fd644: Status 404 returned error can't find the container with id 5fcf880eb94426993d275bd8a98c8aa4669c7140aeb00f6a609030671b6fd644 Oct 30 01:45:00.896221 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.896195 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs"] Oct 30 01:45:00.896581 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:45:00.896552 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f54c97f_7992_47fd_8695_9405fbceb7a7.slice/crio-01708ef036bd9345c7ebea2fb22fadef2df8ab8c4db113204ea3e66300aa9919 WatchSource:0}: Error finding container 01708ef036bd9345c7ebea2fb22fadef2df8ab8c4db113204ea3e66300aa9919: Status 404 returned error can't find the container with id 01708ef036bd9345c7ebea2fb22fadef2df8ab8c4db113204ea3e66300aa9919 Oct 30 01:45:00.899646 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.899627 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9"] Oct 30 01:45:00.942654 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:45:00.942627 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffae23ed_c896_4b25_89cd_759b22580968.slice/crio-49f0a8a60aa8753aa70ab42c84cce07c51e6cc6859edb1e669f3b05b06a3c21d WatchSource:0}: Error finding container 49f0a8a60aa8753aa70ab42c84cce07c51e6cc6859edb1e669f3b05b06a3c21d: Status 404 returned error can't find the container with id 49f0a8a60aa8753aa70ab42c84cce07c51e6cc6859edb1e669f3b05b06a3c21d Oct 30 01:45:00.944734 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:00.944714 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk"] Oct 30 01:45:01.006135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:01.006088 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" event={"ID":"9a0d887c-1b6b-48f8-8f24-4e1f31599281","Type":"ContainerStarted","Data":"00281a8de008af926ddffd50f548e4dbddcf45084f60dbf48b2bb57942496595"} Oct 30 01:45:01.007467 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:01.007445 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" event={"ID":"ffae23ed-c896-4b25-89cd-759b22580968","Type":"ContainerStarted","Data":"49f0a8a60aa8753aa70ab42c84cce07c51e6cc6859edb1e669f3b05b06a3c21d"} Oct 30 01:45:01.008796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:01.008776 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" event={"ID":"4f54c97f-7992-47fd-8695-9405fbceb7a7","Type":"ContainerStarted","Data":"01708ef036bd9345c7ebea2fb22fadef2df8ab8c4db113204ea3e66300aa9919"} Oct 30 01:45:01.010040 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:01.010018 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" event={"ID":"7e963382-3d1d-4cd1-8b63-ecab9d057801","Type":"ContainerStarted","Data":"5fcf880eb94426993d275bd8a98c8aa4669c7140aeb00f6a609030671b6fd644"} Oct 30 01:45:02.019053 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:02.019018 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" event={"ID":"9a0d887c-1b6b-48f8-8f24-4e1f31599281","Type":"ContainerStarted","Data":"366221be15dd79529c440b86cbc1f4b85b3b8de6c5e0f99a9d1ae9d8eb1bd7dd"} Oct 30 01:45:02.020775 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:02.020744 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" event={"ID":"ffae23ed-c896-4b25-89cd-759b22580968","Type":"ContainerStarted","Data":"f2266a27b060e2b8d034b50d7171516a8a13b24a62410a6c9fa41fcfd239b3fc"} Oct 30 01:45:02.022245 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:02.022224 5055 generic.go:356] "Generic (PLEG): container finished" podID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerID="63f52aaf6d5f02533e0f9a3614a3f47c7792cb093df85b245fe887cc330133c5" exitCode=60 Oct 30 01:45:02.022400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:02.022362 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" event={"ID":"4f54c97f-7992-47fd-8695-9405fbceb7a7","Type":"ContainerDied","Data":"63f52aaf6d5f02533e0f9a3614a3f47c7792cb093df85b245fe887cc330133c5"} Oct 30 01:45:02.022683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:02.022665 5055 scope.go:122] "RemoveContainer" containerID="63f52aaf6d5f02533e0f9a3614a3f47c7792cb093df85b245fe887cc330133c5" Oct 30 01:45:02.024159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:02.024109 5055 generic.go:356] "Generic (PLEG): container finished" podID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerID="a831733b31c30af141668f4918762be74d8284439477f8310a44f5b1c90c472a" exitCode=60 Oct 30 01:45:02.024159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:02.024147 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" event={"ID":"7e963382-3d1d-4cd1-8b63-ecab9d057801","Type":"ContainerDied","Data":"a831733b31c30af141668f4918762be74d8284439477f8310a44f5b1c90c472a"} Oct 30 01:45:02.024445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:02.024431 5055 scope.go:122] "RemoveContainer" containerID="a831733b31c30af141668f4918762be74d8284439477f8310a44f5b1c90c472a" Oct 30 01:45:02.078528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:02.078487 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" podStartSLOduration=2.078474196 podStartE2EDuration="2.078474196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.07648554 +0000 UTC m=+1669.397442577" watchObservedRunningTime="2026-10-30 01:45:02.078474196 +0000 UTC m=+1669.399431234" Oct 30 01:45:02.184930 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:02.184887 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" podStartSLOduration=2.184874033 podStartE2EDuration="2.184874033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.18451215 +0000 UTC m=+1669.505469187" watchObservedRunningTime="2026-10-30 01:45:02.184874033 +0000 UTC m=+1669.505831071" Oct 30 01:45:03.031776 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.031745 5055 generic.go:356] "Generic (PLEG): container finished" podID="ffae23ed-c896-4b25-89cd-759b22580968" containerID="f2266a27b060e2b8d034b50d7171516a8a13b24a62410a6c9fa41fcfd239b3fc" exitCode=60 Oct 30 01:45:03.032183 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.031798 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" event={"ID":"ffae23ed-c896-4b25-89cd-759b22580968","Type":"ContainerDied","Data":"f2266a27b060e2b8d034b50d7171516a8a13b24a62410a6c9fa41fcfd239b3fc"} Oct 30 01:45:03.032183 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.032126 5055 scope.go:122] "RemoveContainer" containerID="f2266a27b060e2b8d034b50d7171516a8a13b24a62410a6c9fa41fcfd239b3fc" Oct 30 01:45:03.034043 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.034017 5055 generic.go:356] "Generic (PLEG): container finished" podID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerID="eab7ed7acd16378920c3ac908b1d2228c7cb2872729ee0f97ff07a00a0fe37c9" exitCode=60 Oct 30 01:45:03.034143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.034059 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" event={"ID":"4f54c97f-7992-47fd-8695-9405fbceb7a7","Type":"ContainerDied","Data":"eab7ed7acd16378920c3ac908b1d2228c7cb2872729ee0f97ff07a00a0fe37c9"} Oct 30 01:45:03.034143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.034112 5055 scope.go:122] "RemoveContainer" containerID="63f52aaf6d5f02533e0f9a3614a3f47c7792cb093df85b245fe887cc330133c5" Oct 30 01:45:03.034389 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.034374 5055 scope.go:122] "RemoveContainer" containerID="eab7ed7acd16378920c3ac908b1d2228c7cb2872729ee0f97ff07a00a0fe37c9" Oct 30 01:45:03.035156 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:03.035132 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9_odh-ai-gateway-infra(4f54c97f-7992-47fd-8695-9405fbceb7a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" Oct 30 01:45:03.035956 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.035937 5055 generic.go:356] "Generic (PLEG): container finished" podID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerID="391a9b74e38e8aca831e7110f32e3bd71801b89a2bbe1c75126ab9a31db97f59" exitCode=60 Oct 30 01:45:03.036048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.036013 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" event={"ID":"7e963382-3d1d-4cd1-8b63-ecab9d057801","Type":"ContainerDied","Data":"391a9b74e38e8aca831e7110f32e3bd71801b89a2bbe1c75126ab9a31db97f59"} Oct 30 01:45:03.036251 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.036236 5055 scope.go:122] "RemoveContainer" containerID="391a9b74e38e8aca831e7110f32e3bd71801b89a2bbe1c75126ab9a31db97f59" Oct 30 01:45:03.036493 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:03.036474 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888745-6l2cs_odh-ai-gateway-infra(7e963382-3d1d-4cd1-8b63-ecab9d057801)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" Oct 30 01:45:03.037641 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.037626 5055 generic.go:356] "Generic (PLEG): container finished" podID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerID="366221be15dd79529c440b86cbc1f4b85b3b8de6c5e0f99a9d1ae9d8eb1bd7dd" exitCode=60 Oct 30 01:45:03.037724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.037649 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" event={"ID":"9a0d887c-1b6b-48f8-8f24-4e1f31599281","Type":"ContainerDied","Data":"366221be15dd79529c440b86cbc1f4b85b3b8de6c5e0f99a9d1ae9d8eb1bd7dd"} Oct 30 01:45:03.037885 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.037872 5055 scope.go:122] "RemoveContainer" containerID="366221be15dd79529c440b86cbc1f4b85b3b8de6c5e0f99a9d1ae9d8eb1bd7dd" Oct 30 01:45:03.099225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:03.099202 5055 scope.go:122] "RemoveContainer" containerID="a831733b31c30af141668f4918762be74d8284439477f8310a44f5b1c90c472a" Oct 30 01:45:04.046018 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:04.045990 5055 scope.go:122] "RemoveContainer" containerID="eab7ed7acd16378920c3ac908b1d2228c7cb2872729ee0f97ff07a00a0fe37c9" Oct 30 01:45:04.046439 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:04.046293 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9_odh-ai-gateway-infra(4f54c97f-7992-47fd-8695-9405fbceb7a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" Oct 30 01:45:04.047815 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:04.047787 5055 scope.go:122] "RemoveContainer" containerID="391a9b74e38e8aca831e7110f32e3bd71801b89a2bbe1c75126ab9a31db97f59" Oct 30 01:45:04.048089 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:04.048068 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888745-6l2cs_odh-ai-gateway-infra(7e963382-3d1d-4cd1-8b63-ecab9d057801)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" Oct 30 01:45:04.049560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:04.049528 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" event={"ID":"9a0d887c-1b6b-48f8-8f24-4e1f31599281","Type":"ContainerStarted","Data":"b547187806d827aea94e619edcefd88cc3161c63bfd9a62c60d100804fd3e7b7"} Oct 30 01:45:04.051703 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:04.051685 5055 generic.go:356] "Generic (PLEG): container finished" podID="ffae23ed-c896-4b25-89cd-759b22580968" containerID="c5b87af04cb72ec3027355b965d3a629399bb88ef1917210397891de4170f933" exitCode=60 Oct 30 01:45:04.051791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:04.051744 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" event={"ID":"ffae23ed-c896-4b25-89cd-759b22580968","Type":"ContainerDied","Data":"c5b87af04cb72ec3027355b965d3a629399bb88ef1917210397891de4170f933"} Oct 30 01:45:04.051791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:04.051782 5055 scope.go:122] "RemoveContainer" containerID="f2266a27b060e2b8d034b50d7171516a8a13b24a62410a6c9fa41fcfd239b3fc" Oct 30 01:45:04.051968 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:04.051948 5055 scope.go:122] "RemoveContainer" containerID="c5b87af04cb72ec3027355b965d3a629399bb88ef1917210397891de4170f933" Oct 30 01:45:04.052205 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:04.052183 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk_odh-ai-gateway-infra(ffae23ed-c896-4b25-89cd-759b22580968)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" podUID="ffae23ed-c896-4b25-89cd-759b22580968" Oct 30 01:45:05.059214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:05.059180 5055 generic.go:356] "Generic (PLEG): container finished" podID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerID="b547187806d827aea94e619edcefd88cc3161c63bfd9a62c60d100804fd3e7b7" exitCode=60 Oct 30 01:45:05.059763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:05.059421 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" event={"ID":"9a0d887c-1b6b-48f8-8f24-4e1f31599281","Type":"ContainerDied","Data":"b547187806d827aea94e619edcefd88cc3161c63bfd9a62c60d100804fd3e7b7"} Oct 30 01:45:05.059763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:05.059488 5055 scope.go:122] "RemoveContainer" containerID="366221be15dd79529c440b86cbc1f4b85b3b8de6c5e0f99a9d1ae9d8eb1bd7dd" Oct 30 01:45:05.059763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:05.059571 5055 scope.go:122] "RemoveContainer" containerID="b547187806d827aea94e619edcefd88cc3161c63bfd9a62c60d100804fd3e7b7" Oct 30 01:45:05.059949 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:05.059910 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94_odh-ai-gateway-infra(9a0d887c-1b6b-48f8-8f24-4e1f31599281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" Oct 30 01:45:05.061600 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:05.061586 5055 scope.go:122] "RemoveContainer" containerID="c5b87af04cb72ec3027355b965d3a629399bb88ef1917210397891de4170f933" Oct 30 01:45:05.061832 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:05.061807 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk_odh-ai-gateway-infra(ffae23ed-c896-4b25-89cd-759b22580968)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" podUID="ffae23ed-c896-4b25-89cd-759b22580968" Oct 30 01:45:06.071129 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:06.071096 5055 scope.go:122] "RemoveContainer" containerID="b547187806d827aea94e619edcefd88cc3161c63bfd9a62c60d100804fd3e7b7" Oct 30 01:45:06.071527 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:06.071483 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94_odh-ai-gateway-infra(9a0d887c-1b6b-48f8-8f24-4e1f31599281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" Oct 30 01:45:13.063798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.062699 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl"] Oct 30 01:45:13.077089 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.077059 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.134544 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.134513 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl"] Oct 30 01:45:13.232417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.232385 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.232616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.232453 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.232616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.232474 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.232747 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.232635 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.232806 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.232744 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmxl\" (UniqueName: \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-kube-api-access-lqmxl\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.232867 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.232851 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.232924 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.232871 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.232924 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.232894 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.232924 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.232912 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334236 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334163 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334236 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334236 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmxl\" (UniqueName: \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-kube-api-access-lqmxl\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334292 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334311 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334357 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334373 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334396 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334434 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334457 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334857 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334825 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334905 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.334979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334943 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.335024 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.334972 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.335194 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.335176 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.336971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.336947 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.337118 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.337103 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.354730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.354705 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.354991 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.354971 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmxl\" (UniqueName: \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-kube-api-access-lqmxl\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.455489 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.455461 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:13.794934 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:45:13.794906 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee3e48ad_a4ca_418f_843e_46d87af6ee94.slice/crio-7fbab3e3335069009b35662f76fc6c50ba5e9ed5dbbe8cdb02d940b2fe9dd364 WatchSource:0}: Error finding container 7fbab3e3335069009b35662f76fc6c50ba5e9ed5dbbe8cdb02d940b2fe9dd364: Status 404 returned error can't find the container with id 7fbab3e3335069009b35662f76fc6c50ba5e9ed5dbbe8cdb02d940b2fe9dd364 Oct 30 01:45:13.797913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.797879 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:13.797999 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.797944 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:13.797999 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.797981 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:13.798583 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.798555 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl"] Oct 30 01:45:13.812379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:13.812320 5055 scope.go:122] "RemoveContainer" containerID="391a9b74e38e8aca831e7110f32e3bd71801b89a2bbe1c75126ab9a31db97f59" Oct 30 01:45:14.173218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:14.173186 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" event={"ID":"ee3e48ad-a4ca-418f-843e-46d87af6ee94","Type":"ContainerStarted","Data":"6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6"} Oct 30 01:45:14.173656 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:14.173248 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" event={"ID":"ee3e48ad-a4ca-418f-843e-46d87af6ee94","Type":"ContainerStarted","Data":"7fbab3e3335069009b35662f76fc6c50ba5e9ed5dbbe8cdb02d940b2fe9dd364"} Oct 30 01:45:14.225263 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:14.225207 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" podStartSLOduration=1.225190549 podStartE2EDuration="1.225190549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:14.222670838 +0000 UTC m=+1681.543627876" watchObservedRunningTime="2026-10-30 01:45:14.225190549 +0000 UTC m=+1681.546147593" Oct 30 01:45:14.460345 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:14.460283 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:45:14.460526 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:14.460364 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 01:45:14.464007 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:14.460684 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:15.289351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:15.276537 5055 generic.go:356] "Generic (PLEG): container finished" podID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerID="c2dd5abcb53f5b37b6df9a26fbdabeefb76d8f1b6028eabfc6c0ecf34444ff94" exitCode=60 Oct 30 01:45:15.289351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:15.277524 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" event={"ID":"7e963382-3d1d-4cd1-8b63-ecab9d057801","Type":"ContainerDied","Data":"c2dd5abcb53f5b37b6df9a26fbdabeefb76d8f1b6028eabfc6c0ecf34444ff94"} Oct 30 01:45:15.289351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:15.277593 5055 scope.go:122] "RemoveContainer" containerID="391a9b74e38e8aca831e7110f32e3bd71801b89a2bbe1c75126ab9a31db97f59" Oct 30 01:45:15.289351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:15.277987 5055 scope.go:122] "RemoveContainer" containerID="c2dd5abcb53f5b37b6df9a26fbdabeefb76d8f1b6028eabfc6c0ecf34444ff94" Oct 30 01:45:15.289351 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:15.278239 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888745-6l2cs_odh-ai-gateway-infra(7e963382-3d1d-4cd1-8b63-ecab9d057801)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" Oct 30 01:45:15.465377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:15.464659 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:45:15.465377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:15.464720 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 01:45:16.355352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.349602 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:16.367348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.363867 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs"] Oct 30 01:45:16.437507 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.437458 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:16.462367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.456727 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:45:16.462367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.456788 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 01:45:16.473360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.467396 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:16.557999 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.557424 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:16.583358 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.582598 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:16.798276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.789395 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:45:16.980347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.979463 5055 scope.go:122] "RemoveContainer" containerID="c5b87af04cb72ec3027355b965d3a629399bb88ef1917210397891de4170f933" Oct 30 01:45:16.980347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.980284 5055 scope.go:122] "RemoveContainer" containerID="eab7ed7acd16378920c3ac908b1d2228c7cb2872729ee0f97ff07a00a0fe37c9" Oct 30 01:45:17.029400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:16.980922 5055 scope.go:122] "RemoveContainer" containerID="b547187806d827aea94e619edcefd88cc3161c63bfd9a62c60d100804fd3e7b7" Oct 30 01:45:17.057353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.033458 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:17.358027 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.357997 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" Oct 30 01:45:17.413939 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.413907 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" event={"ID":"7e963382-3d1d-4cd1-8b63-ecab9d057801","Type":"ContainerDied","Data":"5fcf880eb94426993d275bd8a98c8aa4669c7140aeb00f6a609030671b6fd644"} Oct 30 01:45:17.414174 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.414157 5055 scope.go:122] "RemoveContainer" containerID="c2dd5abcb53f5b37b6df9a26fbdabeefb76d8f1b6028eabfc6c0ecf34444ff94" Oct 30 01:45:17.414440 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.414422 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs" Oct 30 01:45:17.459147 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.457205 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e963382-3d1d-4cd1-8b63-ecab9d057801-kube-api-access-mln9m\" (UniqueName: \"kubernetes.io/projected/7e963382-3d1d-4cd1-8b63-ecab9d057801-kube-api-access-mln9m\") pod \"7e963382-3d1d-4cd1-8b63-ecab9d057801\" (UID: \"7e963382-3d1d-4cd1-8b63-ecab9d057801\") " Oct 30 01:45:17.459147 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.457309 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e963382-3d1d-4cd1-8b63-ecab9d057801-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e963382-3d1d-4cd1-8b63-ecab9d057801-maas-api-service-ca\") pod \"7e963382-3d1d-4cd1-8b63-ecab9d057801\" (UID: \"7e963382-3d1d-4cd1-8b63-ecab9d057801\") " Oct 30 01:45:17.459147 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.457773 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:45:17.459147 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.457817 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 01:45:17.459147 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.458852 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e963382-3d1d-4cd1-8b63-ecab9d057801-maas-api-service-ca" pod "7e963382-3d1d-4cd1-8b63-ecab9d057801" (UID: "7e963382-3d1d-4cd1-8b63-ecab9d057801"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:17.460002 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.459959 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e963382-3d1d-4cd1-8b63-ecab9d057801-maas-api-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:45:17.487723 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.478455 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e963382-3d1d-4cd1-8b63-ecab9d057801-kube-api-access-mln9m" pod "7e963382-3d1d-4cd1-8b63-ecab9d057801" (UID: "7e963382-3d1d-4cd1-8b63-ecab9d057801"). InnerVolumeSpecName "kube-api-access-mln9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:17.560805 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.560739 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mln9m\" (UniqueName: \"kubernetes.io/projected/7e963382-3d1d-4cd1-8b63-ecab9d057801-kube-api-access-mln9m\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:45:17.760294 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.760266 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs"] Oct 30 01:45:17.789619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:17.789597 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888745-6l2cs"] Oct 30 01:45:18.423013 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:18.422984 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" event={"ID":"4f54c97f-7992-47fd-8695-9405fbceb7a7","Type":"ContainerStarted","Data":"abd951c72b1e587ebd8eea897095b1b15a4da4d336698df468dfd15de1670b96"} Oct 30 01:45:18.426103 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:18.426079 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" event={"ID":"9a0d887c-1b6b-48f8-8f24-4e1f31599281","Type":"ContainerStarted","Data":"cd0d9725b999b23ae904ee5d8f43a7dde58741b4640aff277a2e1455cfbb9d37"} Oct 30 01:45:18.427912 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:18.427889 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" event={"ID":"ffae23ed-c896-4b25-89cd-759b22580968","Type":"ContainerStarted","Data":"0ec35fa090dc212a0822a74cd0332e59d1a22b1ee5cf18013025d36ad408413b"} Oct 30 01:45:18.456557 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:18.456536 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:45:18.456640 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:18.456580 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 01:45:18.504137 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:18.504011 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" podStartSLOduration=18.503993874 podStartE2EDuration="18.503993874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:18.475846544 +0000 UTC m=+1685.796803584" watchObservedRunningTime="2026-10-30 01:45:18.503993874 +0000 UTC m=+1685.824950914" Oct 30 01:45:18.820200 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:18.820171 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" path="/var/lib/kubelet/pods/7e963382-3d1d-4cd1-8b63-ecab9d057801/volumes" Oct 30 01:45:19.159368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.159274 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv"] Oct 30 01:45:19.160173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.160152 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerName="cleanup" Oct 30 01:45:19.160279 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.160267 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerName="cleanup" Oct 30 01:45:19.160380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.160372 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerName="cleanup" Oct 30 01:45:19.160448 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.160440 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerName="cleanup" Oct 30 01:45:19.160645 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.160636 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerName="cleanup" Oct 30 01:45:19.160712 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.160706 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerName="cleanup" Oct 30 01:45:19.186348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.180771 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.220206 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.218971 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv"] Oct 30 01:45:19.220206 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.219770 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q57f5\" (UniqueName: \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-kube-api-access-q57f5\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.220551 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.220526 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.220697 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.220680 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.220887 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.220867 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.222182 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.221994 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.222182 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.222032 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced403d9-79de-4102-be8a-753038340507-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.222182 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.222079 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.222446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.222243 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced403d9-79de-4102-be8a-753038340507-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.222446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.222287 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.235914 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:19.235885 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:45:19.323340 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.323301 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced403d9-79de-4102-be8a-753038340507-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.323496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.323365 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.323746 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.323726 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.323798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.323782 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q57f5\" (UniqueName: \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-kube-api-access-q57f5\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.323870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.323827 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.323870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.323848 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.323970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.323914 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.324023 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.323961 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.324023 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.323996 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced403d9-79de-4102-be8a-753038340507-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.324138 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.324031 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced403d9-79de-4102-be8a-753038340507-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.324138 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.324044 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.324222 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.324206 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.324277 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.324255 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.324312 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.324298 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.326505 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.326484 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced403d9-79de-4102-be8a-753038340507-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.326623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.326532 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.345958 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.345934 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.346497 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.346478 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q57f5\" (UniqueName: \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-kube-api-access-q57f5\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.436289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.436222 5055 generic.go:356] "Generic (PLEG): container finished" podID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerID="cd0d9725b999b23ae904ee5d8f43a7dde58741b4640aff277a2e1455cfbb9d37" exitCode=60 Oct 30 01:45:19.436794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.436291 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" event={"ID":"9a0d887c-1b6b-48f8-8f24-4e1f31599281","Type":"ContainerDied","Data":"cd0d9725b999b23ae904ee5d8f43a7dde58741b4640aff277a2e1455cfbb9d37"} Oct 30 01:45:19.436794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.436361 5055 scope.go:122] "RemoveContainer" containerID="b547187806d827aea94e619edcefd88cc3161c63bfd9a62c60d100804fd3e7b7" Oct 30 01:45:19.436794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.436720 5055 scope.go:122] "RemoveContainer" containerID="cd0d9725b999b23ae904ee5d8f43a7dde58741b4640aff277a2e1455cfbb9d37" Oct 30 01:45:19.437024 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:19.437005 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94_odh-ai-gateway-infra(9a0d887c-1b6b-48f8-8f24-4e1f31599281)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" Oct 30 01:45:19.438151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.438132 5055 generic.go:356] "Generic (PLEG): container finished" podID="ffae23ed-c896-4b25-89cd-759b22580968" containerID="0ec35fa090dc212a0822a74cd0332e59d1a22b1ee5cf18013025d36ad408413b" exitCode=60 Oct 30 01:45:19.438245 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.438201 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" event={"ID":"ffae23ed-c896-4b25-89cd-759b22580968","Type":"ContainerDied","Data":"0ec35fa090dc212a0822a74cd0332e59d1a22b1ee5cf18013025d36ad408413b"} Oct 30 01:45:19.438595 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.438578 5055 scope.go:122] "RemoveContainer" containerID="0ec35fa090dc212a0822a74cd0332e59d1a22b1ee5cf18013025d36ad408413b" Oct 30 01:45:19.438844 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:19.438823 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk_odh-ai-gateway-infra(ffae23ed-c896-4b25-89cd-759b22580968)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" podUID="ffae23ed-c896-4b25-89cd-759b22580968" Oct 30 01:45:19.440637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.440616 5055 generic.go:356] "Generic (PLEG): container finished" podID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerID="abd951c72b1e587ebd8eea897095b1b15a4da4d336698df468dfd15de1670b96" exitCode=60 Oct 30 01:45:19.440813 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.440753 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" event={"ID":"4f54c97f-7992-47fd-8695-9405fbceb7a7","Type":"ContainerDied","Data":"abd951c72b1e587ebd8eea897095b1b15a4da4d336698df468dfd15de1670b96"} Oct 30 01:45:19.441206 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.441182 5055 scope.go:122] "RemoveContainer" containerID="abd951c72b1e587ebd8eea897095b1b15a4da4d336698df468dfd15de1670b96" Oct 30 01:45:19.441593 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:19.441557 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9_odh-ai-gateway-infra(4f54c97f-7992-47fd-8695-9405fbceb7a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" Oct 30 01:45:19.451975 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.451959 5055 scope.go:122] "RemoveContainer" containerID="c5b87af04cb72ec3027355b965d3a629399bb88ef1917210397891de4170f933" Oct 30 01:45:19.456669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.456635 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:45:19.456744 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.456688 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 01:45:19.501474 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.501263 5055 scope.go:122] "RemoveContainer" containerID="eab7ed7acd16378920c3ac908b1d2228c7cb2872729ee0f97ff07a00a0fe37c9" Oct 30 01:45:19.507669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.507647 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9"] Oct 30 01:45:19.515515 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.515499 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:19.534249 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.534170 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94"] Oct 30 01:45:19.620506 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:19.620387 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk"] Oct 30 01:45:20.035873 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:45:20.035838 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced403d9_79de_4102_be8a_753038340507.slice/crio-3c675628717f4cb9a52a9a6ea2a7c55b5dbba5ac441359d3a6624477a12474b5 WatchSource:0}: Error finding container 3c675628717f4cb9a52a9a6ea2a7c55b5dbba5ac441359d3a6624477a12474b5: Status 404 returned error can't find the container with id 3c675628717f4cb9a52a9a6ea2a7c55b5dbba5ac441359d3a6624477a12474b5 Oct 30 01:45:20.038957 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.038916 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:20.039071 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.039002 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:20.039071 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.039041 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:20.039904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.039884 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv"] Oct 30 01:45:20.448832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.448757 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" event={"ID":"ced403d9-79de-4102-be8a-753038340507","Type":"ContainerStarted","Data":"8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853"} Oct 30 01:45:20.448832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.448805 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" event={"ID":"ced403d9-79de-4102-be8a-753038340507","Type":"ContainerStarted","Data":"3c675628717f4cb9a52a9a6ea2a7c55b5dbba5ac441359d3a6624477a12474b5"} Oct 30 01:45:20.455819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.455786 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:45:20.455911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.455856 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 01:45:20.523288 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.522110 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:20.524744 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.524688 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" podStartSLOduration=1.524670365 podStartE2EDuration="1.524670365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:20.521859751 +0000 UTC m=+1687.842816790" watchObservedRunningTime="2026-10-30 01:45:20.524670365 +0000 UTC m=+1687.845627396" Oct 30 01:45:20.751057 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.751032 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" Oct 30 01:45:20.869205 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.869177 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" Oct 30 01:45:20.873713 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.873687 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" Oct 30 01:45:20.914262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.914230 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffae23ed-c896-4b25-89cd-759b22580968-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffae23ed-c896-4b25-89cd-759b22580968-maas-api-service-ca\") pod \"ffae23ed-c896-4b25-89cd-759b22580968\" (UID: \"ffae23ed-c896-4b25-89cd-759b22580968\") " Oct 30 01:45:20.914439 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.914294 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffae23ed-c896-4b25-89cd-759b22580968-kube-api-access-5h5jv\" (UniqueName: \"kubernetes.io/projected/ffae23ed-c896-4b25-89cd-759b22580968-kube-api-access-5h5jv\") pod \"ffae23ed-c896-4b25-89cd-759b22580968\" (UID: \"ffae23ed-c896-4b25-89cd-759b22580968\") " Oct 30 01:45:20.914626 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.914605 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffae23ed-c896-4b25-89cd-759b22580968-maas-api-service-ca" pod "ffae23ed-c896-4b25-89cd-759b22580968" (UID: "ffae23ed-c896-4b25-89cd-759b22580968"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:20.914814 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.914796 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffae23ed-c896-4b25-89cd-759b22580968-maas-api-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:45:20.916747 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:20.916727 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffae23ed-c896-4b25-89cd-759b22580968-kube-api-access-5h5jv" pod "ffae23ed-c896-4b25-89cd-759b22580968" (UID: "ffae23ed-c896-4b25-89cd-759b22580968"). InnerVolumeSpecName "kube-api-access-5h5jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:21.016247 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.016145 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0d887c-1b6b-48f8-8f24-4e1f31599281-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a0d887c-1b6b-48f8-8f24-4e1f31599281-maas-api-service-ca\") pod \"9a0d887c-1b6b-48f8-8f24-4e1f31599281\" (UID: \"9a0d887c-1b6b-48f8-8f24-4e1f31599281\") " Oct 30 01:45:21.016448 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.016280 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f54c97f-7992-47fd-8695-9405fbceb7a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f54c97f-7992-47fd-8695-9405fbceb7a7-maas-api-service-ca\") pod \"4f54c97f-7992-47fd-8695-9405fbceb7a7\" (UID: \"4f54c97f-7992-47fd-8695-9405fbceb7a7\") " Oct 30 01:45:21.016448 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.016413 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f54c97f-7992-47fd-8695-9405fbceb7a7-kube-api-access-zgds6\" (UniqueName: \"kubernetes.io/projected/4f54c97f-7992-47fd-8695-9405fbceb7a7-kube-api-access-zgds6\") pod \"4f54c97f-7992-47fd-8695-9405fbceb7a7\" (UID: \"4f54c97f-7992-47fd-8695-9405fbceb7a7\") " Oct 30 01:45:21.016584 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.016486 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0d887c-1b6b-48f8-8f24-4e1f31599281-kube-api-access-rlv9w\" (UniqueName: \"kubernetes.io/projected/9a0d887c-1b6b-48f8-8f24-4e1f31599281-kube-api-access-rlv9w\") pod \"9a0d887c-1b6b-48f8-8f24-4e1f31599281\" (UID: \"9a0d887c-1b6b-48f8-8f24-4e1f31599281\") " Oct 30 01:45:21.017058 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.017038 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h5jv\" (UniqueName: \"kubernetes.io/projected/ffae23ed-c896-4b25-89cd-759b22580968-kube-api-access-5h5jv\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:45:21.021047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.021019 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f54c97f-7992-47fd-8695-9405fbceb7a7-maas-api-service-ca" pod "4f54c97f-7992-47fd-8695-9405fbceb7a7" (UID: "4f54c97f-7992-47fd-8695-9405fbceb7a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:21.021263 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.021243 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0d887c-1b6b-48f8-8f24-4e1f31599281-maas-api-service-ca" pod "9a0d887c-1b6b-48f8-8f24-4e1f31599281" (UID: "9a0d887c-1b6b-48f8-8f24-4e1f31599281"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:21.022694 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.022673 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0d887c-1b6b-48f8-8f24-4e1f31599281-kube-api-access-rlv9w" pod "9a0d887c-1b6b-48f8-8f24-4e1f31599281" (UID: "9a0d887c-1b6b-48f8-8f24-4e1f31599281"). InnerVolumeSpecName "kube-api-access-rlv9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:21.023378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.023359 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f54c97f-7992-47fd-8695-9405fbceb7a7-kube-api-access-zgds6" pod "4f54c97f-7992-47fd-8695-9405fbceb7a7" (UID: "4f54c97f-7992-47fd-8695-9405fbceb7a7"). InnerVolumeSpecName "kube-api-access-zgds6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:21.119386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.118541 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a0d887c-1b6b-48f8-8f24-4e1f31599281-maas-api-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:45:21.119386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.118568 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f54c97f-7992-47fd-8695-9405fbceb7a7-maas-api-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:45:21.119386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.118578 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgds6\" (UniqueName: \"kubernetes.io/projected/4f54c97f-7992-47fd-8695-9405fbceb7a7-kube-api-access-zgds6\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:45:21.119386 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.118587 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlv9w\" (UniqueName: \"kubernetes.io/projected/9a0d887c-1b6b-48f8-8f24-4e1f31599281-kube-api-access-rlv9w\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:45:21.456151 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.456116 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:45:21.456583 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.456165 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 01:45:21.474357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.474140 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" event={"ID":"9a0d887c-1b6b-48f8-8f24-4e1f31599281","Type":"ContainerDied","Data":"00281a8de008af926ddffd50f548e4dbddcf45084f60dbf48b2bb57942496595"} Oct 30 01:45:21.474357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.474230 5055 scope.go:122] "RemoveContainer" containerID="cd0d9725b999b23ae904ee5d8f43a7dde58741b4640aff277a2e1455cfbb9d37" Oct 30 01:45:21.474626 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.474426 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94" Oct 30 01:45:21.478513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.478488 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" event={"ID":"ffae23ed-c896-4b25-89cd-759b22580968","Type":"ContainerDied","Data":"49f0a8a60aa8753aa70ab42c84cce07c51e6cc6859edb1e669f3b05b06a3c21d"} Oct 30 01:45:21.478730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.478710 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk" Oct 30 01:45:21.480902 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.480868 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" Oct 30 01:45:21.482588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.482383 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9" event={"ID":"4f54c97f-7992-47fd-8695-9405fbceb7a7","Type":"ContainerDied","Data":"01708ef036bd9345c7ebea2fb22fadef2df8ab8c4db113204ea3e66300aa9919"} Oct 30 01:45:21.517445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.517263 5055 scope.go:122] "RemoveContainer" containerID="0ec35fa090dc212a0822a74cd0332e59d1a22b1ee5cf18013025d36ad408413b" Oct 30 01:45:21.517863 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.517837 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:21.517982 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.517885 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" podUID="ced403d9-79de-4102-be8a-753038340507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 01:45:21.531914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.531891 5055 scope.go:122] "RemoveContainer" containerID="abd951c72b1e587ebd8eea897095b1b15a4da4d336698df468dfd15de1670b96" Oct 30 01:45:21.598466 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.598378 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94"] Oct 30 01:45:21.607250 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.607215 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-558327-29888745-7sx94"] Oct 30 01:45:21.641947 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.641911 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9"] Oct 30 01:45:21.647977 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.647953 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a3a2ac-29888745-c6xt9"] Oct 30 01:45:21.705465 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.705422 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk"] Oct 30 01:45:21.709143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:21.709115 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-0ae258-29888745-72vmk"] Oct 30 01:45:22.516356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:22.516120 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:22.516356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:22.516174 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" podUID="ced403d9-79de-4102-be8a-753038340507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 01:45:22.632951 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:22.632921 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:22.633845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:22.633823 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:22.634771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:22.634752 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:45:22.828026 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:22.827956 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" path="/var/lib/kubelet/pods/4f54c97f-7992-47fd-8695-9405fbceb7a7/volumes" Oct 30 01:45:22.832060 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:22.832029 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" path="/var/lib/kubelet/pods/9a0d887c-1b6b-48f8-8f24-4e1f31599281/volumes" Oct 30 01:45:22.845864 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:22.845826 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffae23ed-c896-4b25-89cd-759b22580968" path="/var/lib/kubelet/pods/ffae23ed-c896-4b25-89cd-759b22580968/volumes" Oct 30 01:45:23.515859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:23.515829 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:23.516092 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:23.515883 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" podUID="ced403d9-79de-4102-be8a-753038340507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 01:45:24.515792 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:24.515754 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:24.516175 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:24.515825 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" podUID="ced403d9-79de-4102-be8a-753038340507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 01:45:25.516083 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:25.516051 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:25.516495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:25.516103 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" podUID="ced403d9-79de-4102-be8a-753038340507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 01:45:26.516629 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:26.516598 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:26.517137 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:26.516648 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" podUID="ced403d9-79de-4102-be8a-753038340507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 01:45:27.644395 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:27.643469 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:27.645072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:27.644968 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:27.645969 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:27.645949 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:45:35.355577 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.355545 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78"] Oct 30 01:45:35.356099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356085 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ffae23ed-c896-4b25-89cd-759b22580968" containerName="cleanup" Oct 30 01:45:35.356146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356099 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffae23ed-c896-4b25-89cd-759b22580968" containerName="cleanup" Oct 30 01:45:35.356146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356113 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ffae23ed-c896-4b25-89cd-759b22580968" containerName="cleanup" Oct 30 01:45:35.356146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356118 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffae23ed-c896-4b25-89cd-759b22580968" containerName="cleanup" Oct 30 01:45:35.356146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356125 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerName="cleanup" Oct 30 01:45:35.356146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356130 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356150 5055 cpu_manager.go:414] "containerMap: removing container" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356154 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356163 5055 cpu_manager.go:414] "containerMap: removing container" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356167 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356173 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ffae23ed-c896-4b25-89cd-759b22580968" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356178 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffae23ed-c896-4b25-89cd-759b22580968" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356193 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356197 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356210 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356214 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356286 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffae23ed-c896-4b25-89cd-759b22580968" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356292 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerName="cleanup" Oct 30 01:45:35.356301 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356302 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerName="cleanup" Oct 30 01:45:35.356682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356308 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerName="cleanup" Oct 30 01:45:35.356682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356313 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e963382-3d1d-4cd1-8b63-ecab9d057801" containerName="cleanup" Oct 30 01:45:35.356682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356319 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffae23ed-c896-4b25-89cd-759b22580968" containerName="cleanup" Oct 30 01:45:35.356682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.356342 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerName="cleanup" Oct 30 01:45:35.359847 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.359815 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.399792 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.399755 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78"] Oct 30 01:45:35.492659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.492615 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.492840 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.492774 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.492921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.492846 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.492921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.492874 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.493037 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.492960 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.493037 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.492995 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.493105 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.493036 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptr8l\" (UniqueName: \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-kube-api-access-ptr8l\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.493142 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.493103 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.493142 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.493128 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.594382 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.594344 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.594382 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.594387 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.594621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.594405 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.594621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.594441 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.594621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.594468 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.594621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.594521 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptr8l\" (UniqueName: \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-kube-api-access-ptr8l\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.594621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.594614 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.594890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.594652 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.594890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.594716 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.595171 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.595151 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.595246 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.595224 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.595366 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.595321 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.595454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.595433 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.595544 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.595531 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.597385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.597361 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.597622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.597602 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.613702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.613644 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.613863 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.613841 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptr8l\" (UniqueName: \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-kube-api-access-ptr8l\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.676050 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.676027 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:35.991039 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:45:35.991009 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8186c325_cf8c_4a7b_b0ac_e92058eb18ce.slice/crio-6a799d5810f94a1af2dbc22ed77a8f877b5a1d679caf66ba062588e49662789e WatchSource:0}: Error finding container 6a799d5810f94a1af2dbc22ed77a8f877b5a1d679caf66ba062588e49662789e: Status 404 returned error can't find the container with id 6a799d5810f94a1af2dbc22ed77a8f877b5a1d679caf66ba062588e49662789e Oct 30 01:45:35.992254 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.992230 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78"] Oct 30 01:45:35.993607 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.993566 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:35.993705 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.993648 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:35.993705 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:35.993690 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:36.626425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:36.626384 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" event={"ID":"8186c325-cf8c-4a7b-b0ac-e92058eb18ce","Type":"ContainerStarted","Data":"6bbf4d5d00f532c448667b4b8b135b5769ab6af83472e4d5bbffcc2f27cb4bfa"} Oct 30 01:45:36.626860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:36.626445 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" event={"ID":"8186c325-cf8c-4a7b-b0ac-e92058eb18ce","Type":"ContainerStarted","Data":"6a799d5810f94a1af2dbc22ed77a8f877b5a1d679caf66ba062588e49662789e"} Oct 30 01:45:36.680350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:36.676424 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:36.692439 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:36.692373 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" podStartSLOduration=1.692305983 podStartE2EDuration="1.692305983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:36.689245679 +0000 UTC m=+1704.010202721" watchObservedRunningTime="2026-10-30 01:45:36.692305983 +0000 UTC m=+1704.013263025" Oct 30 01:45:37.680468 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:37.680435 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:37.681028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:37.680996 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 01:45:38.711349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:38.707715 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:38.715455 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:38.715419 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 01:45:39.680164 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:39.680123 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:39.680476 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:39.680445 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 01:45:40.681350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:40.680720 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:40.681350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:40.680793 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 01:45:41.688352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:41.680641 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:41.688352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:41.680720 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 01:45:42.676746 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:42.676712 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:42.676926 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:42.676808 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 01:45:43.154123 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.154093 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd"] Oct 30 01:45:43.154685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.154670 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerName="cleanup" Oct 30 01:45:43.154685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.154684 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerName="cleanup" Oct 30 01:45:43.154835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.154709 5055 cpu_manager.go:414] "containerMap: removing container" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerName="cleanup" Oct 30 01:45:43.154835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.154713 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerName="cleanup" Oct 30 01:45:43.154835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.154792 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f54c97f-7992-47fd-8695-9405fbceb7a7" containerName="cleanup" Oct 30 01:45:43.154835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.154800 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffae23ed-c896-4b25-89cd-759b22580968" containerName="cleanup" Oct 30 01:45:43.154835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.154812 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0d887c-1b6b-48f8-8f24-4e1f31599281" containerName="cleanup" Oct 30 01:45:43.158341 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.158312 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.201648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.201613 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd"] Oct 30 01:45:43.209399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.209372 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.209635 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.209613 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.209800 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.209781 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.209971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.209955 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.210155 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.210129 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfcf547-3b5f-4907-b88e-392969c79158-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.210249 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.210172 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfcf547-3b5f-4907-b88e-392969c79158-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.210249 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.210225 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.210380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.210260 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.210380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.210313 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4bs\" (UniqueName: \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-kube-api-access-tc4bs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.311816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.311773 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfcf547-3b5f-4907-b88e-392969c79158-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.312008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.311824 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfcf547-3b5f-4907-b88e-392969c79158-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.312008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.311875 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.312008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.311909 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.312008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.311962 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4bs\" (UniqueName: \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-kube-api-access-tc4bs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.312214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.312119 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.312214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.312162 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.312310 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.312211 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.312310 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.312282 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.312979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.312919 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.312979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.312944 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.313156 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.313051 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.313427 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.313398 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfcf547-3b5f-4907-b88e-392969c79158-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.313676 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.313655 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.315600 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.315576 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.316176 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.316153 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfcf547-3b5f-4907-b88e-392969c79158-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.333616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.333590 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.334417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.334396 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4bs\" (UniqueName: \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-kube-api-access-tc4bs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.489675 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.489636 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:43.822858 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.822779 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd"] Oct 30 01:45:43.834318 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:45:43.834286 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfcf547_3b5f_4907_b88e_392969c79158.slice/crio-372508e8b88f3aa75720baf35f03a995e4f28e028f1c04d3f72de12b01308a9a WatchSource:0}: Error finding container 372508e8b88f3aa75720baf35f03a995e4f28e028f1c04d3f72de12b01308a9a: Status 404 returned error can't find the container with id 372508e8b88f3aa75720baf35f03a995e4f28e028f1c04d3f72de12b01308a9a Oct 30 01:45:43.837128 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.837088 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:43.837204 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.837178 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:43.837250 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:43.837219 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:44.170947 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.170451 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" event={"ID":"dcfcf547-3b5f-4907-b88e-392969c79158","Type":"ContainerStarted","Data":"d4c68c868f28f4b9ea4e54a20bb39b80244e3faf8ed35345b61f37b4f1f58090"} Oct 30 01:45:44.170947 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.170519 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" event={"ID":"dcfcf547-3b5f-4907-b88e-392969c79158","Type":"ContainerStarted","Data":"372508e8b88f3aa75720baf35f03a995e4f28e028f1c04d3f72de12b01308a9a"} Oct 30 01:45:44.244715 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.244640 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" podStartSLOduration=1.244622472 podStartE2EDuration="1.244622472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:44.230347559 +0000 UTC m=+1711.551304592" watchObservedRunningTime="2026-10-30 01:45:44.244622472 +0000 UTC m=+1711.565579512" Oct 30 01:45:44.495801 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.493581 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:44.499468 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.499437 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:45:44.516354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.508036 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 01:45:44.732378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.729032 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:45:44.732378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.729093 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:44.900381 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.899696 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:44.902359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.901144 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:44.905150 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:44.905129 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:45:45.493351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:45.492736 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:45:45.493351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:45.492795 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 01:45:46.490814 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:46.490764 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:45:46.491112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:46.491087 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 01:45:47.491062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:47.491022 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:45:47.491826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:47.491790 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 01:45:48.490447 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:48.490398 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:45:48.490639 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:48.490473 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 01:45:49.250452 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:45:49.250413 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:45:49.490794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:49.490758 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:45:49.491071 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:49.491047 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 01:45:50.421148 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.421118 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t"] Oct 30 01:45:50.426619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.426598 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.447157 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.447127 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t"] Oct 30 01:45:50.490735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.490705 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 01:45:50.490991 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.490958 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 01:45:50.567288 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.567248 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.567483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.567306 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.567483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.567438 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.567483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.567472 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.567671 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.567546 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.567671 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.567604 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.567671 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.567631 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68rw2\" (UniqueName: \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-kube-api-access-68rw2\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.567837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.567707 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.567837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.567760 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.668788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.668749 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.669015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.668825 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.669015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.668863 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.669015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.668897 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.669015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.668957 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.669239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.669036 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.669239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.669079 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68rw2\" (UniqueName: \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-kube-api-access-68rw2\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.669239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.669145 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.669878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.669423 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.669878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.669818 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.670223 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.670202 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.670356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.670282 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.670588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.670562 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.671528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.671479 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.671682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.671666 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.671807 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.671791 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.689359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.689304 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.689626 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.689605 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68rw2\" (UniqueName: \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-kube-api-access-68rw2\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:50.745150 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:50.745119 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:51.086182 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:45:51.086146 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice/crio-06e9ba175e62c5bf4be74c1a1336ca52c77b43d3f1a8e54d46f0ceb091f1faed WatchSource:0}: Error finding container 06e9ba175e62c5bf4be74c1a1336ca52c77b43d3f1a8e54d46f0ceb091f1faed: Status 404 returned error can't find the container with id 06e9ba175e62c5bf4be74c1a1336ca52c77b43d3f1a8e54d46f0ceb091f1faed Oct 30 01:45:51.086579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.086557 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t"] Oct 30 01:45:51.089225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.089192 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:51.089367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.089252 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:51.089367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.089279 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:45:51.243740 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.243704 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" event={"ID":"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94","Type":"ContainerStarted","Data":"e836f07493e78d9562c862fca1fda8c46b525612268a93aee9caf9548c0e4d72"} Oct 30 01:45:51.243909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.243779 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" event={"ID":"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94","Type":"ContainerStarted","Data":"06e9ba175e62c5bf4be74c1a1336ca52c77b43d3f1a8e54d46f0ceb091f1faed"} Oct 30 01:45:51.289203 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.289141 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" podStartSLOduration=1.289122691 podStartE2EDuration="1.289122691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:51.286589335 +0000 UTC m=+1718.607546375" watchObservedRunningTime="2026-10-30 01:45:51.289122691 +0000 UTC m=+1718.610079731" Oct 30 01:45:51.495999 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.495972 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:51.496502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.496369 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:51.497193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.497176 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:45:51.745657 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:51.745629 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:52.747141 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:52.747105 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:52.747586 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:52.747158 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 01:45:53.746190 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:53.746143 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:53.746400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:53.746213 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 01:45:54.746146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:54.746111 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:54.746564 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:54.746165 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 01:45:55.745593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:55.745560 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:55.745767 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:55.745613 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 01:45:56.745717 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:56.745679 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:56.746360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:56.746309 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 01:45:57.745862 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:57.745824 5055 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:57.746378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:57.745894 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" Oct 30 01:45:58.751068 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:58.751037 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:58.751581 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:58.751307 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:45:58.752010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:45:58.751998 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:46:01.351473 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:01.351417 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.394433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:01.394386 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 01:46:01.433531 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:01.433490 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:46:01.507473 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:01.507426 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.575513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:01.575468 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.575863 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:01.575835 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded" Oct 30 01:46:01.788603 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:01.788555 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.828555 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:01.827423 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.930529 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:01.930483 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.931081 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:01.931053 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded" Oct 30 01:46:02.068562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.068454 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.364422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.360426 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.364422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.361152 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:46:02.461357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.436968 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 01:46:02.461357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.437720 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:46:02.507415 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.497456 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 01:46:02.507415 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.497961 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:46:02.723187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.687808 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:46:02.723187 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.692819 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:46:02.779354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.734856 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Oct 30 01:46:02.779646 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.744013 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.817390 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.789508 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:46:02.817390 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.790485 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Oct 30 01:46:02.817390 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.791588 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:46:02.853086 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.853042 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.951399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.940800 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.963357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.952998 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:46:02.963357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:02.953055 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:46:03.081215 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.081121 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.107992 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.107964 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:46:03.394633 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.367049 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 01:46:03.449619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.449487 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.508539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.508495 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.691560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.691426 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.798370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.783426 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.798370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.796463 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.865355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.854602 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.881433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.881405 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:46:03.941368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:03.941317 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:04.050670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:04.050640 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 01:46:04.111463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:04.111423 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:04.170189 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:04.170157 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 01:46:04.179036 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:04.179009 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 01:46:04.274382 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:04.273618 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:46:04.389669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:04.388615 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:46:04.422429 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:04.422304 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:46:04.449814 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:04.447574 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:46:04.739020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:04.719923 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:46:17.374480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:17.374424 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.445504 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:17.445457 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.526250 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:17.526207 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.766953 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:17.766916 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.808465 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:17.808411 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.810087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:17.810060 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.850503 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:17.850464 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.863734 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:17.863403 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 01:46:18.115638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:18.115529 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:18.472204 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:18.472155 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded" Oct 30 01:46:18.474114 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:18.474080 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:46:18.585354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:18.425497 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 01:46:18.585354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:18.585104 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:46:18.843404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:18.767453 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:19.073351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:18.810575 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Oct 30 01:46:19.098398 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:18.883684 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:19.098398 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.077435 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:19.119349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.119286 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:19.164367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.155644 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:46:19.181848 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.181822 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:46:19.183144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.183125 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:46:19.183356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.183341 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:46:19.271657 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.271629 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 01:46:19.271921 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.196924 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:19.478357 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.473412 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:19.569115 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.569053 5055 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:19.569659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.569594 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.62:8000/ready\": context deadline exceeded" Oct 30 01:46:19.770710 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.770620 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 01:46:19.889359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.880790 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 01:46:19.902360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.899416 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 01:46:19.927788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.927764 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 01:46:19.939495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:19.939155 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 01:46:19.942267 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:46:19.942242 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:46:21.007720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.007686 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh"] Oct 30 01:46:21.008208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.007927 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" podUID="80ac42ae-8948-4f15-ab7a-66717bb47fc2" containerName="maas-api" containerID="cri-o://0330cd61df61478418012e85c32bac39ccbdd4dd51da5a61941a1d78fdf6a333" gracePeriod=30 Oct 30 01:46:21.603303 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.603164 5055 generic.go:356] "Generic (PLEG): container finished" podID="80ac42ae-8948-4f15-ab7a-66717bb47fc2" containerID="0330cd61df61478418012e85c32bac39ccbdd4dd51da5a61941a1d78fdf6a333" exitCode=0 Oct 30 01:46:21.603303 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.603240 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" event={"ID":"80ac42ae-8948-4f15-ab7a-66717bb47fc2","Type":"ContainerDied","Data":"0330cd61df61478418012e85c32bac39ccbdd4dd51da5a61941a1d78fdf6a333"} Oct 30 01:46:21.836661 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.836599 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:46:21.899486 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.899448 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-maas-api-tls\") pod \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " Oct 30 01:46:21.899717 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.899696 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ac42ae-8948-4f15-ab7a-66717bb47fc2-kube-api-access-hxkmt\" (UniqueName: \"kubernetes.io/projected/80ac42ae-8948-4f15-ab7a-66717bb47fc2-kube-api-access-hxkmt\") pod \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " Oct 30 01:46:21.899798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.899744 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-metrics-certs\") pod \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " Oct 30 01:46:21.899857 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.899817 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ac42ae-8948-4f15-ab7a-66717bb47fc2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ac42ae-8948-4f15-ab7a-66717bb47fc2-openshift-service-ca\") pod \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\" (UID: \"80ac42ae-8948-4f15-ab7a-66717bb47fc2\") " Oct 30 01:46:21.900531 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.900504 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ac42ae-8948-4f15-ab7a-66717bb47fc2-openshift-service-ca" pod "80ac42ae-8948-4f15-ab7a-66717bb47fc2" (UID: "80ac42ae-8948-4f15-ab7a-66717bb47fc2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:21.906457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.906432 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ac42ae-8948-4f15-ab7a-66717bb47fc2-kube-api-access-hxkmt" pod "80ac42ae-8948-4f15-ab7a-66717bb47fc2" (UID: "80ac42ae-8948-4f15-ab7a-66717bb47fc2"). InnerVolumeSpecName "kube-api-access-hxkmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:21.910930 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.910899 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-metrics-certs" pod "80ac42ae-8948-4f15-ab7a-66717bb47fc2" (UID: "80ac42ae-8948-4f15-ab7a-66717bb47fc2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:21.912653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:21.912625 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-maas-api-tls" pod "80ac42ae-8948-4f15-ab7a-66717bb47fc2" (UID: "80ac42ae-8948-4f15-ab7a-66717bb47fc2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:22.001737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:22.001700 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxkmt\" (UniqueName: \"kubernetes.io/projected/80ac42ae-8948-4f15-ab7a-66717bb47fc2-kube-api-access-hxkmt\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:46:22.001737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:22.001730 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:46:22.001737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:22.001739 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ac42ae-8948-4f15-ab7a-66717bb47fc2-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:46:22.001955 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:22.001751 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ac42ae-8948-4f15-ab7a-66717bb47fc2-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:46:22.611242 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:22.611162 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" Oct 30 01:46:22.611729 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:22.611173 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh" event={"ID":"80ac42ae-8948-4f15-ab7a-66717bb47fc2","Type":"ContainerDied","Data":"00b51e618ba5ee878af81e3f355e2642f3e53e615e4252ebc42ff4c7e6fe437f"} Oct 30 01:46:22.611729 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:22.611340 5055 scope.go:122] "RemoveContainer" containerID="0330cd61df61478418012e85c32bac39ccbdd4dd51da5a61941a1d78fdf6a333" Oct 30 01:46:22.663556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:22.663525 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh"] Oct 30 01:46:22.675781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:22.675756 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-0ae258-57dbb6d967-mzfbh"] Oct 30 01:46:22.822645 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:22.822614 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ac42ae-8948-4f15-ab7a-66717bb47fc2" path="/var/lib/kubelet/pods/80ac42ae-8948-4f15-ab7a-66717bb47fc2/volumes" Oct 30 01:46:28.288407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.288371 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn"] Oct 30 01:46:28.289009 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.288994 5055 cpu_manager.go:414] "containerMap: removing container" podUID="80ac42ae-8948-4f15-ab7a-66717bb47fc2" containerName="maas-api" Oct 30 01:46:28.289051 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.289011 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ac42ae-8948-4f15-ab7a-66717bb47fc2" containerName="maas-api" Oct 30 01:46:28.289141 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.289132 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ac42ae-8948-4f15-ab7a-66717bb47fc2" containerName="maas-api" Oct 30 01:46:28.293803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.293786 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.329482 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.329447 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn"] Oct 30 01:46:28.402748 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.402718 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.402923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.402894 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.402965 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.402932 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.402965 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.402953 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.403100 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.403080 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlncs\" (UniqueName: \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-kube-api-access-zlncs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.403168 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.403151 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.403203 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.403190 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.403305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.403288 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.403407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.403339 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.504276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504239 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.504461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504290 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.504461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504320 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.504461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504406 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlncs\" (UniqueName: \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-kube-api-access-zlncs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.504461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504454 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.504752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504487 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.504752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504555 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.504752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504579 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.504752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504632 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.504752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504700 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.505010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.504929 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.505177 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.505159 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.505404 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.505383 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.505498 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.505478 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.506821 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.506802 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.507166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.507148 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.536397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.536368 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlncs\" (UniqueName: \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-kube-api-access-zlncs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.536612 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.536589 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:28.621727 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:28.621640 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:29.079747 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:29.075387 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn"] Oct 30 01:46:29.080345 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:29.080277 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:46:29.080710 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:29.080686 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:46:29.081002 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:29.080866 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:46:29.668751 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:29.668617 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" event={"ID":"ea0b9039-8705-4ba7-8942-991b5fd80d9b","Type":"ContainerStarted","Data":"b3f3fc8af7c7ebfb7b61426f2dc53a146dfc1454907dd67bacf27490a146ee83"} Oct 30 01:46:29.669593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:29.669564 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" event={"ID":"ea0b9039-8705-4ba7-8942-991b5fd80d9b","Type":"ContainerStarted","Data":"5ddcf1710a3ff60b2b479e62afa69ac7ed996aae832d1c892475ea2aec829ade"} Oct 30 01:46:29.755302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:29.755252 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" podStartSLOduration=1.7552379230000001 podStartE2EDuration="1.755237923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:29.740088801 +0000 UTC m=+1757.061045839" watchObservedRunningTime="2026-10-30 01:46:29.755237923 +0000 UTC m=+1757.076194961" Oct 30 01:46:30.622094 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:30.622061 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:30.623757 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:30.623727 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:46:30.623878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:30.623789 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 01:46:31.628131 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:31.628097 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:46:31.628736 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:31.628709 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 01:46:32.623183 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:32.623144 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:46:32.623422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:32.623393 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 01:46:33.622699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:33.622659 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:46:33.623090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:33.622709 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 01:46:34.622270 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:34.622235 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:46:34.622458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:34.622291 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 01:46:35.622719 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:35.622685 5055 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:46:35.623119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:35.622759 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" Oct 30 01:46:36.627578 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:36.627547 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:36.628051 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:36.627906 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:36.628477 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:36.628457 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:46:39.575179 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.575136 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz"] Oct 30 01:46:39.579217 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.579196 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:39.601066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.600994 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz"] Oct 30 01:46:39.650033 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.650001 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 01:46:39.650212 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.650125 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 01:46:39.713141 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.713088 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:39.713381 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.713209 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:39.713381 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.713318 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5lx\" (UniqueName: \"kubernetes.io/projected/5bbee30a-0f1e-4492-9ebc-b646050ff492-kube-api-access-lh5lx\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:39.814483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.814449 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:39.814699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.814521 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:39.814699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.814641 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5lx\" (UniqueName: \"kubernetes.io/projected/5bbee30a-0f1e-4492-9ebc-b646050ff492-kube-api-access-lh5lx\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:39.815185 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.815162 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:39.817095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.817076 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:39.841583 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.841553 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5lx\" (UniqueName: \"kubernetes.io/projected/5bbee30a-0f1e-4492-9ebc-b646050ff492-kube-api-access-lh5lx\") pod \"maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:39.893230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:39.893133 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:40.205610 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:40.205576 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz"] Oct 30 01:46:40.205785 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:46:40.205729 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbee30a_0f1e_4492_9ebc_b646050ff492.slice/crio-866c3181c04995a535dfa0119b563fd9d9e086b53f8a46cc5cc3bf0adaca1a57 WatchSource:0}: Error finding container 866c3181c04995a535dfa0119b563fd9d9e086b53f8a46cc5cc3bf0adaca1a57: Status 404 returned error can't find the container with id 866c3181c04995a535dfa0119b563fd9d9e086b53f8a46cc5cc3bf0adaca1a57 Oct 30 01:46:40.899681 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:40.899566 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" event={"ID":"5bbee30a-0f1e-4492-9ebc-b646050ff492","Type":"ContainerStarted","Data":"44bcd2874b736e0ec2ff87d79cafe68fb6b2a272afbcf2543c7a258e461e867d"} Oct 30 01:46:40.902488 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:40.902459 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" event={"ID":"5bbee30a-0f1e-4492-9ebc-b646050ff492","Type":"ContainerStarted","Data":"866c3181c04995a535dfa0119b563fd9d9e086b53f8a46cc5cc3bf0adaca1a57"} Oct 30 01:46:40.952984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:40.952932 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" podStartSLOduration=1.952914517 podStartE2EDuration="1.952914517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:40.938367767 +0000 UTC m=+1768.259324809" watchObservedRunningTime="2026-10-30 01:46:40.952914517 +0000 UTC m=+1768.273871559" Oct 30 01:46:41.942352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:41.940848 5055 generic.go:356] "Generic (PLEG): container finished" podID="5bbee30a-0f1e-4492-9ebc-b646050ff492" containerID="44bcd2874b736e0ec2ff87d79cafe68fb6b2a272afbcf2543c7a258e461e867d" exitCode=0 Oct 30 01:46:41.942352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:41.940965 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" event={"ID":"5bbee30a-0f1e-4492-9ebc-b646050ff492","Type":"ContainerDied","Data":"44bcd2874b736e0ec2ff87d79cafe68fb6b2a272afbcf2543c7a258e461e867d"} Oct 30 01:46:43.584795 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.584763 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:43.743236 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.743205 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bbee30a-0f1e-4492-9ebc-b646050ff492-kube-api-access-lh5lx\" (UniqueName: \"kubernetes.io/projected/5bbee30a-0f1e-4492-9ebc-b646050ff492-kube-api-access-lh5lx\") pod \"5bbee30a-0f1e-4492-9ebc-b646050ff492\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " Oct 30 01:46:43.743460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.743346 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-openshift-ca\") pod \"5bbee30a-0f1e-4492-9ebc-b646050ff492\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " Oct 30 01:46:43.743460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.743407 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-xks-ca\") pod \"5bbee30a-0f1e-4492-9ebc-b646050ff492\" (UID: \"5bbee30a-0f1e-4492-9ebc-b646050ff492\") " Oct 30 01:46:43.743817 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.743790 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-openshift-ca" pod "5bbee30a-0f1e-4492-9ebc-b646050ff492" (UID: "5bbee30a-0f1e-4492-9ebc-b646050ff492"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:43.743977 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.743964 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-openshift-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.745506 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.745474 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bbee30a-0f1e-4492-9ebc-b646050ff492-kube-api-access-lh5lx" pod "5bbee30a-0f1e-4492-9ebc-b646050ff492" (UID: "5bbee30a-0f1e-4492-9ebc-b646050ff492"). InnerVolumeSpecName "kube-api-access-lh5lx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:43.745625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.745605 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-xks-ca" pod "5bbee30a-0f1e-4492-9ebc-b646050ff492" (UID: "5bbee30a-0f1e-4492-9ebc-b646050ff492"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:43.845226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.845156 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5lx\" (UniqueName: \"kubernetes.io/projected/5bbee30a-0f1e-4492-9ebc-b646050ff492-kube-api-access-lh5lx\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.845226 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.845181 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5bbee30a-0f1e-4492-9ebc-b646050ff492-maas-api-xks-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.982656 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.982627 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" Oct 30 01:46:43.982656 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.982641 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz" event={"ID":"5bbee30a-0f1e-4492-9ebc-b646050ff492","Type":"ContainerDied","Data":"866c3181c04995a535dfa0119b563fd9d9e086b53f8a46cc5cc3bf0adaca1a57"} Oct 30 01:46:43.982907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:43.982682 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866c3181c04995a535dfa0119b563fd9d9e086b53f8a46cc5cc3bf0adaca1a57" Oct 30 01:46:49.218281 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:46:49.218232 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:46:53.648020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:53.647979 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz"] Oct 30 01:46:53.658407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:53.658379 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-8f68ab-69sqz"] Oct 30 01:46:54.820349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:46:54.820301 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bbee30a-0f1e-4492-9ebc-b646050ff492" path="/var/lib/kubelet/pods/5bbee30a-0f1e-4492-9ebc-b646050ff492/volumes" Oct 30 01:47:11.158406 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.158373 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p"] Oct 30 01:47:11.158965 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.158952 5055 cpu_manager.go:414] "containerMap: removing container" podUID="5bbee30a-0f1e-4492-9ebc-b646050ff492" containerName="revoke-keys" Oct 30 01:47:11.159008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.158966 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bbee30a-0f1e-4492-9ebc-b646050ff492" containerName="revoke-keys" Oct 30 01:47:11.159062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.159054 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bbee30a-0f1e-4492-9ebc-b646050ff492" containerName="revoke-keys" Oct 30 01:47:11.163020 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.162996 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.189368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.188683 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-abb30f-serving-cert\"" Oct 30 01:47:11.193816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.193791 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p"] Oct 30 01:47:11.250581 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.250545 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.250766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.250595 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb94ef17-f808-4ec3-8c42-d48160efb641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.250858 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.250835 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmpgq\" (UniqueName: \"kubernetes.io/projected/bb94ef17-f808-4ec3-8c42-d48160efb641-kube-api-access-cmpgq\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.250933 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.250889 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.360368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.355098 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmpgq\" (UniqueName: \"kubernetes.io/projected/bb94ef17-f808-4ec3-8c42-d48160efb641-kube-api-access-cmpgq\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.360368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.355189 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.360368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.355290 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.360368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.355402 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb94ef17-f808-4ec3-8c42-d48160efb641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.360368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.356554 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb94ef17-f808-4ec3-8c42-d48160efb641-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.370274 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.370240 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.373315 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.373289 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.376523 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.376501 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmpgq\" (UniqueName: \"kubernetes.io/projected/bb94ef17-f808-4ec3-8c42-d48160efb641-kube-api-access-cmpgq\") pod \"maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.480001 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.479968 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:11.784814 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:47:11.784750 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb94ef17_f808_4ec3_8c42_d48160efb641.slice/crio-359aca662f6e829b7803333dd167e69c499c3add7b09e4501bf4283f42eda22a WatchSource:0}: Error finding container 359aca662f6e829b7803333dd167e69c499c3add7b09e4501bf4283f42eda22a: Status 404 returned error can't find the container with id 359aca662f6e829b7803333dd167e69c499c3add7b09e4501bf4283f42eda22a Oct 30 01:47:11.786190 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.786174 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:47:11.786904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:11.786762 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p"] Oct 30 01:47:12.398561 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:12.398526 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" event={"ID":"bb94ef17-f808-4ec3-8c42-d48160efb641","Type":"ContainerStarted","Data":"359aca662f6e829b7803333dd167e69c499c3add7b09e4501bf4283f42eda22a"} Oct 30 01:47:13.133576 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:13.133536 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:47:13.137143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:13.137112 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:47:13.406109 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:13.406026 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" event={"ID":"bb94ef17-f808-4ec3-8c42-d48160efb641","Type":"ContainerStarted","Data":"0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1"} Oct 30 01:47:13.406559 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:13.406218 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:13.450079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:13.450040 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" podStartSLOduration=2.009469906 podStartE2EDuration="2.450028609s" totalImagesPullingTime="440.558703ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:13.448941725 +0000 UTC m=+1800.769898763" watchObservedRunningTime="2026-10-30 01:47:13.450028609 +0000 UTC m=+1800.770985647" Oct 30 01:47:13.563444 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:13.563402 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh"] Oct 30 01:47:13.563705 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:13.563680 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" containerID="cri-o://dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd" gracePeriod=30 Oct 30 01:47:18.860408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:18.860357 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:47:19.008881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.008851 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-certs\") pod \"f19a3a45-89c3-4f99-a84a-2645813b36d9\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " Oct 30 01:47:19.009080 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.008917 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-kube-api-access-4jjpw\" (UniqueName: \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-kube-api-access-4jjpw\") pod \"f19a3a45-89c3-4f99-a84a-2645813b36d9\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " Oct 30 01:47:19.009080 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009026 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-credential-socket\") pod \"f19a3a45-89c3-4f99-a84a-2645813b36d9\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " Oct 30 01:47:19.009196 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009080 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-socket\") pod \"f19a3a45-89c3-4f99-a84a-2645813b36d9\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " Oct 30 01:47:19.009196 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009136 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-envoy\") pod \"f19a3a45-89c3-4f99-a84a-2645813b36d9\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " Oct 30 01:47:19.009196 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009171 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-certs" pod "f19a3a45-89c3-4f99-a84a-2645813b36d9" (UID: "f19a3a45-89c3-4f99-a84a-2645813b36d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:19.009196 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009193 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19a3a45-89c3-4f99-a84a-2645813b36d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19a3a45-89c3-4f99-a84a-2645813b36d9-istiod-ca-cert\") pod \"f19a3a45-89c3-4f99-a84a-2645813b36d9\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " Oct 30 01:47:19.009445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009251 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-data\") pod \"f19a3a45-89c3-4f99-a84a-2645813b36d9\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " Oct 30 01:47:19.009445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009305 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-socket" pod "f19a3a45-89c3-4f99-a84a-2645813b36d9" (UID: "f19a3a45-89c3-4f99-a84a-2645813b36d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:19.009445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009344 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-credential-socket" pod "f19a3a45-89c3-4f99-a84a-2645813b36d9" (UID: "f19a3a45-89c3-4f99-a84a-2645813b36d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:19.009445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009317 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-token\" (UniqueName: \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-token\") pod \"f19a3a45-89c3-4f99-a84a-2645813b36d9\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " Oct 30 01:47:19.009445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009429 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-podinfo\") pod \"f19a3a45-89c3-4f99-a84a-2645813b36d9\" (UID: \"f19a3a45-89c3-4f99-a84a-2645813b36d9\") " Oct 30 01:47:19.009698 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009477 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-data" pod "f19a3a45-89c3-4f99-a84a-2645813b36d9" (UID: "f19a3a45-89c3-4f99-a84a-2645813b36d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:19.009698 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.009600 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19a3a45-89c3-4f99-a84a-2645813b36d9-istiod-ca-cert" pod "f19a3a45-89c3-4f99-a84a-2645813b36d9" (UID: "f19a3a45-89c3-4f99-a84a-2645813b36d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:47:19.010286 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.010261 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:19.010449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.010290 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:19.010449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.010305 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:19.010449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.010318 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:19.010449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.010348 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f19a3a45-89c3-4f99-a84a-2645813b36d9-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:19.012094 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.012069 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-envoy" pod "f19a3a45-89c3-4f99-a84a-2645813b36d9" (UID: "f19a3a45-89c3-4f99-a84a-2645813b36d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:19.012460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.012438 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-kube-api-access-4jjpw" pod "f19a3a45-89c3-4f99-a84a-2645813b36d9" (UID: "f19a3a45-89c3-4f99-a84a-2645813b36d9"). InnerVolumeSpecName "kube-api-access-4jjpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:19.012543 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.012488 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-token" pod "f19a3a45-89c3-4f99-a84a-2645813b36d9" (UID: "f19a3a45-89c3-4f99-a84a-2645813b36d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:19.012606 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.012582 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-podinfo" pod "f19a3a45-89c3-4f99-a84a-2645813b36d9" (UID: "f19a3a45-89c3-4f99-a84a-2645813b36d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:47:19.112265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.112188 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jjpw\" (UniqueName: \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-kube-api-access-4jjpw\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:19.112265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.112226 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:19.112265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.112243 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:19.112265 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.112255 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f19a3a45-89c3-4f99-a84a-2645813b36d9-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:19.218228 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:47:19.218194 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:47:19.418437 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.418367 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 01:47:19.453766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.453735 5055 generic.go:356] "Generic (PLEG): container finished" podID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerID="dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd" exitCode=0 Oct 30 01:47:19.453928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.453779 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" event={"ID":"f19a3a45-89c3-4f99-a84a-2645813b36d9","Type":"ContainerDied","Data":"dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd"} Oct 30 01:47:19.453928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.453803 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" Oct 30 01:47:19.453928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.453831 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh" event={"ID":"f19a3a45-89c3-4f99-a84a-2645813b36d9","Type":"ContainerDied","Data":"ac7248aeb2b99ece59675554cc53390accb242ad2d8c6e0f9c81ffce81ebe864"} Oct 30 01:47:19.453928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.453871 5055 scope.go:122] "RemoveContainer" containerID="dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd" Oct 30 01:47:19.466339 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.466288 5055 scope.go:122] "RemoveContainer" containerID="349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b" Oct 30 01:47:19.475851 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.475825 5055 scope.go:122] "RemoveContainer" containerID="dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd" Oct 30 01:47:19.476279 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:47:19.476254 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd\": container with ID starting with dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd not found: ID does not exist" containerID="dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd" Oct 30 01:47:19.476395 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.476290 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd"} err="failed to get container status \"dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd\": rpc error: code = NotFound desc = could not find container \"dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd\": container with ID starting with dd1ea97e93d8abed2aa446cd9e5338a3da8427e43c0c41fc7b40fb3d8db656dd not found: ID does not exist" Oct 30 01:47:19.476395 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.476317 5055 scope.go:122] "RemoveContainer" containerID="349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b" Oct 30 01:47:19.476648 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:47:19.476631 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b\": container with ID starting with 349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b not found: ID does not exist" containerID="349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b" Oct 30 01:47:19.476718 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.476654 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b"} err="failed to get container status \"349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b\": rpc error: code = NotFound desc = could not find container \"349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b\": container with ID starting with 349e66dcefb61321671403e1788b8f14c105ae26060230e3b7365de6e6f6e63b not found: ID does not exist" Oct 30 01:47:19.520612 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.520580 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh"] Oct 30 01:47:19.526045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:19.526002 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-0ae258-openshift-default-7f9d7575cc-tdqbh"] Oct 30 01:47:20.791784 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.791753 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p"] Oct 30 01:47:20.792422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.792408 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" Oct 30 01:47:20.792469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.792423 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" Oct 30 01:47:20.792469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.792444 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" Oct 30 01:47:20.792469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.792453 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" Oct 30 01:47:20.792588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.792574 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" Oct 30 01:47:20.792632 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.792594 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" containerName="istio-proxy" Oct 30 01:47:20.797150 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.797129 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:20.826067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.826033 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19a3a45-89c3-4f99-a84a-2645813b36d9" path="/var/lib/kubelet/pods/f19a3a45-89c3-4f99-a84a-2645813b36d9/volumes" Oct 30 01:47:20.855715 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.855684 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-b6b7fc-serving-cert\"" Oct 30 01:47:20.860730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.860702 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p"] Oct 30 01:47:20.957597 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.957549 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee10591-c2b7-4290-a568-e6d39436988f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:20.957755 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.957732 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr6g7\" (UniqueName: \"kubernetes.io/projected/fee10591-c2b7-4290-a568-e6d39436988f-kube-api-access-cr6g7\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:20.957861 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.957846 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:20.957899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:20.957874 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:21.059746 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:21.059655 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:21.059746 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:21.059693 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:21.060039 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:21.059763 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee10591-c2b7-4290-a568-e6d39436988f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:21.060039 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:47:21.059838 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-serving-cert: secret "maas-api-e2e-shared-a-w2-b6b7fc-serving-cert" not found Oct 30 01:47:21.060039 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:21.059916 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr6g7\" (UniqueName: \"kubernetes.io/projected/fee10591-c2b7-4290-a568-e6d39436988f-kube-api-access-cr6g7\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:21.060039 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:47:21.059937 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-maas-api-tls podName:fee10591-c2b7-4290-a568-e6d39436988f nodeName:}" failed. No retries permitted until 2026-10-30 01:47:21.559915554 +0000 UTC m=+1808.880872585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-maas-api-tls") pod "maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" (UID: "fee10591-c2b7-4290-a568-e6d39436988f") : secret "maas-api-e2e-shared-a-w2-b6b7fc-serving-cert" not found Oct 30 01:47:21.060636 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:21.060613 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee10591-c2b7-4290-a568-e6d39436988f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:21.062599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:21.062578 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:21.080260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:21.080235 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr6g7\" (UniqueName: \"kubernetes.io/projected/fee10591-c2b7-4290-a568-e6d39436988f-kube-api-access-cr6g7\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:21.566815 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:21.566773 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:21.573800 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:21.573770 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:21.715364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:21.711458 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:22.177408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:22.171465 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p"] Oct 30 01:47:22.477976 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:22.477942 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" event={"ID":"fee10591-c2b7-4290-a568-e6d39436988f","Type":"ContainerStarted","Data":"45d492e764243467efa83278d3ce06bcdbaf57eb0af014772ec61a806a500d3c"} Oct 30 01:47:23.487771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:23.487731 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" event={"ID":"fee10591-c2b7-4290-a568-e6d39436988f","Type":"ContainerStarted","Data":"d0efa95b5ba90944a45f532ac2b06ab0ba3a2594a331113db174103a7a746707"} Oct 30 01:47:23.488254 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:23.487878 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:23.535483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:23.535426 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" podStartSLOduration=3.053967071 podStartE2EDuration="3.535408147s" totalImagesPullingTime="481.441076ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:23.532042328 +0000 UTC m=+1810.852999367" watchObservedRunningTime="2026-10-30 01:47:23.535408147 +0000 UTC m=+1810.856365186" Oct 30 01:47:26.188209 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.186762 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7"] Oct 30 01:47:26.191895 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.191874 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.219496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.219470 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-c8d376-serving-cert\"" Oct 30 01:47:26.223481 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.223456 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7"] Oct 30 01:47:26.310008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.309977 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f07429-1516-4228-85f1-1c37e3100ad7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.310259 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.310233 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qsx\" (UniqueName: \"kubernetes.io/projected/b6f07429-1516-4228-85f1-1c37e3100ad7-kube-api-access-88qsx\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.310365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.310298 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.310425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.310402 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.411267 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.411235 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qsx\" (UniqueName: \"kubernetes.io/projected/b6f07429-1516-4228-85f1-1c37e3100ad7-kube-api-access-88qsx\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.411468 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.411289 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.411468 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.411346 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.411468 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.411455 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f07429-1516-4228-85f1-1c37e3100ad7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.412125 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.412061 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f07429-1516-4228-85f1-1c37e3100ad7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.413876 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.413852 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.413979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.413917 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.431648 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.431626 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qsx\" (UniqueName: \"kubernetes.io/projected/b6f07429-1516-4228-85f1-1c37e3100ad7-kube-api-access-88qsx\") pod \"maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.507845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.507812 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:26.846825 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:26.845837 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7"] Oct 30 01:47:26.847114 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:47:26.847066 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6f07429_1516_4228_85f1_1c37e3100ad7.slice/crio-d7ddb73f21855145fb37c477b140503afb8e2519be053f3d5abb59c3c89b949d WatchSource:0}: Error finding container d7ddb73f21855145fb37c477b140503afb8e2519be053f3d5abb59c3c89b949d: Status 404 returned error can't find the container with id d7ddb73f21855145fb37c477b140503afb8e2519be053f3d5abb59c3c89b949d Oct 30 01:47:27.551971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:27.551890 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" event={"ID":"b6f07429-1516-4228-85f1-1c37e3100ad7","Type":"ContainerStarted","Data":"2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a"} Oct 30 01:47:27.551971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:27.551965 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" event={"ID":"b6f07429-1516-4228-85f1-1c37e3100ad7","Type":"ContainerStarted","Data":"d7ddb73f21855145fb37c477b140503afb8e2519be053f3d5abb59c3c89b949d"} Oct 30 01:47:27.552899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:27.552877 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:27.595045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:27.594997 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" podStartSLOduration=1.205129468 podStartE2EDuration="1.594984459s" totalImagesPullingTime="389.854991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:27.593967466 +0000 UTC m=+1814.914924499" watchObservedRunningTime="2026-10-30 01:47:27.594984459 +0000 UTC m=+1814.915941499" Oct 30 01:47:29.500287 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:29.500261 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 01:47:29.562049 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:29.562000 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr"] Oct 30 01:47:29.562773 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:29.562730 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerName="istio-proxy" containerID="cri-o://80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8" gracePeriod=30 Oct 30 01:47:34.616381 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.616350 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 01:47:34.885106 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.885077 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:47:34.956172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.956136 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-envoy\") pod \"7b66c7d3-7336-4beb-af15-3c013ec43512\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " Oct 30 01:47:34.956418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.956191 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b66c7d3-7336-4beb-af15-3c013ec43512-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b66c7d3-7336-4beb-af15-3c013ec43512-istiod-ca-cert\") pod \"7b66c7d3-7336-4beb-af15-3c013ec43512\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " Oct 30 01:47:34.956418 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.956319 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-data\") pod \"7b66c7d3-7336-4beb-af15-3c013ec43512\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " Oct 30 01:47:34.956893 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.956823 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b66c7d3-7336-4beb-af15-3c013ec43512-istiod-ca-cert" pod "7b66c7d3-7336-4beb-af15-3c013ec43512" (UID: "7b66c7d3-7336-4beb-af15-3c013ec43512"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:47:34.957123 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.957061 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-data" pod "7b66c7d3-7336-4beb-af15-3c013ec43512" (UID: "7b66c7d3-7336-4beb-af15-3c013ec43512"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:34.957306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.957290 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-kube-api-access-smtkx\" (UniqueName: \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-kube-api-access-smtkx\") pod \"7b66c7d3-7336-4beb-af15-3c013ec43512\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " Oct 30 01:47:34.957456 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.957417 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-podinfo\") pod \"7b66c7d3-7336-4beb-af15-3c013ec43512\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " Oct 30 01:47:34.957548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.957456 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-credential-socket\") pod \"7b66c7d3-7336-4beb-af15-3c013ec43512\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " Oct 30 01:47:34.957548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.957528 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-certs\") pod \"7b66c7d3-7336-4beb-af15-3c013ec43512\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " Oct 30 01:47:34.957702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.957570 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-token\" (UniqueName: \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-token\") pod \"7b66c7d3-7336-4beb-af15-3c013ec43512\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " Oct 30 01:47:34.957702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.957605 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-socket\") pod \"7b66c7d3-7336-4beb-af15-3c013ec43512\" (UID: \"7b66c7d3-7336-4beb-af15-3c013ec43512\") " Oct 30 01:47:34.958166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.958147 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-credential-socket" pod "7b66c7d3-7336-4beb-af15-3c013ec43512" (UID: "7b66c7d3-7336-4beb-af15-3c013ec43512"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:34.958371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.958266 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b66c7d3-7336-4beb-af15-3c013ec43512-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:34.958488 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.958474 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:34.959439 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.958948 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-certs" pod "7b66c7d3-7336-4beb-af15-3c013ec43512" (UID: "7b66c7d3-7336-4beb-af15-3c013ec43512"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:34.959740 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.959719 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-socket" pod "7b66c7d3-7336-4beb-af15-3c013ec43512" (UID: "7b66c7d3-7336-4beb-af15-3c013ec43512"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:34.961366 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.961339 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-envoy" pod "7b66c7d3-7336-4beb-af15-3c013ec43512" (UID: "7b66c7d3-7336-4beb-af15-3c013ec43512"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:34.962313 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.962283 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-token" pod "7b66c7d3-7336-4beb-af15-3c013ec43512" (UID: "7b66c7d3-7336-4beb-af15-3c013ec43512"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:34.962489 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.962467 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-podinfo" pod "7b66c7d3-7336-4beb-af15-3c013ec43512" (UID: "7b66c7d3-7336-4beb-af15-3c013ec43512"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:47:34.963723 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:34.963701 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-kube-api-access-smtkx" pod "7b66c7d3-7336-4beb-af15-3c013ec43512" (UID: "7b66c7d3-7336-4beb-af15-3c013ec43512"). InnerVolumeSpecName "kube-api-access-smtkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:35.059970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.059928 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smtkx\" (UniqueName: \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-kube-api-access-smtkx\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:35.059970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.059961 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:35.059970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.059974 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:35.060225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.059986 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:35.060225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.059998 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:35.060225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.060008 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:35.060225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.060019 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b66c7d3-7336-4beb-af15-3c013ec43512-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:47:35.671798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.671759 5055 generic.go:356] "Generic (PLEG): container finished" podID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerID="80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8" exitCode=0 Oct 30 01:47:35.672306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.671830 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" Oct 30 01:47:35.672306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.671855 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" event={"ID":"7b66c7d3-7336-4beb-af15-3c013ec43512","Type":"ContainerDied","Data":"80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8"} Oct 30 01:47:35.672306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.671932 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr" event={"ID":"7b66c7d3-7336-4beb-af15-3c013ec43512","Type":"ContainerDied","Data":"c4b7e25fbbad4cc4d795138766943354585da455438ca0dd2bccca18f7a46a77"} Oct 30 01:47:35.672306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.671955 5055 scope.go:122] "RemoveContainer" containerID="80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8" Oct 30 01:47:35.695816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.695596 5055 scope.go:122] "RemoveContainer" containerID="80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8" Oct 30 01:47:35.695946 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:47:35.695922 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8\": container with ID starting with 80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8 not found: ID does not exist" containerID="80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8" Oct 30 01:47:35.696014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.695960 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8"} err="failed to get container status \"80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8\": rpc error: code = NotFound desc = could not find container \"80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8\": container with ID starting with 80017fe5f80b03712ee77eb49e01332c603f366bfa7079febf48f25a0a2cdfe8 not found: ID does not exist" Oct 30 01:47:35.738593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.738554 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr"] Oct 30 01:47:35.751970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:35.751870 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-8lpmr"] Oct 30 01:47:36.821866 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:36.821834 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" path="/var/lib/kubelet/pods/7b66c7d3-7336-4beb-af15-3c013ec43512/volumes" Oct 30 01:47:39.154498 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.154467 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v"] Oct 30 01:47:39.155093 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.155079 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerName="istio-proxy" Oct 30 01:47:39.155137 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.155095 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerName="istio-proxy" Oct 30 01:47:39.155194 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.155185 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b66c7d3-7336-4beb-af15-3c013ec43512" containerName="istio-proxy" Oct 30 01:47:39.160451 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.160432 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.187882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.187853 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v"] Oct 30 01:47:39.188031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.187978 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-292018-serving-cert\"" Oct 30 01:47:39.242533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.242501 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.242700 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.242574 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.242700 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.242651 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7381e33-dd3e-4243-818d-21413e7e163e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.242810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.242773 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpvsx\" (UniqueName: \"kubernetes.io/projected/a7381e33-dd3e-4243-818d-21413e7e163e-kube-api-access-kpvsx\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.344546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.344517 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.344707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.344553 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7381e33-dd3e-4243-818d-21413e7e163e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.344707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.344611 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpvsx\" (UniqueName: \"kubernetes.io/projected/a7381e33-dd3e-4243-818d-21413e7e163e-kube-api-access-kpvsx\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.344823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.344742 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.345378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.345286 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7381e33-dd3e-4243-818d-21413e7e163e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.347069 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.347044 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.347173 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.347155 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.365810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.365785 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpvsx\" (UniqueName: \"kubernetes.io/projected/a7381e33-dd3e-4243-818d-21413e7e163e-kube-api-access-kpvsx\") pod \"maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.474524 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.474487 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:39.780062 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:47:39.780031 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7381e33_dd3e_4243_818d_21413e7e163e.slice/crio-c885e54feb658ba73088d9e434df1cf21b4d6a2b0944b1afe8528070f7dd4674 WatchSource:0}: Error finding container c885e54feb658ba73088d9e434df1cf21b4d6a2b0944b1afe8528070f7dd4674: Status 404 returned error can't find the container with id c885e54feb658ba73088d9e434df1cf21b4d6a2b0944b1afe8528070f7dd4674 Oct 30 01:47:39.780645 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:39.780604 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v"] Oct 30 01:47:40.715597 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:40.715563 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" event={"ID":"a7381e33-dd3e-4243-818d-21413e7e163e","Type":"ContainerStarted","Data":"62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac"} Oct 30 01:47:40.715989 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:40.715642 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" event={"ID":"a7381e33-dd3e-4243-818d-21413e7e163e","Type":"ContainerStarted","Data":"c885e54feb658ba73088d9e434df1cf21b4d6a2b0944b1afe8528070f7dd4674"} Oct 30 01:47:40.715989 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:40.715683 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:40.758028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:40.757969 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" podStartSLOduration=1.407147577 podStartE2EDuration="1.757952846s" totalImagesPullingTime="350.805269ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:40.75767902 +0000 UTC m=+1828.078636062" watchObservedRunningTime="2026-10-30 01:47:40.757952846 +0000 UTC m=+1828.078909887" Oct 30 01:47:46.730509 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:46.730478 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 01:47:49.222564 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:47:49.222520 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:47:55.140585 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:55.140388 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr"] Oct 30 01:47:55.141072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:55.140648 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" containerID="cri-o://9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e" gracePeriod=30 Oct 30 01:47:56.686349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.685427 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh"] Oct 30 01:47:56.695349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.693177 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.731878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.731841 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh"] Oct 30 01:47:56.750443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.750417 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-389dba-openshift-default-dockercfg-djv4s\"" Oct 30 01:47:56.786482 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.786269 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4952793b-a587-46ca-bcb5-e9c94e359332-istiod-ca-cert\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.787532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.786696 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-socket\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.787532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.786791 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-istio-token\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.787532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.786948 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nwlm\" (UniqueName: \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-kube-api-access-9nwlm\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.787532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.786983 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-credential-socket\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.787532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.787029 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-data\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.787532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.787152 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-certs\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.787532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.787256 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-envoy\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.787532 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.787290 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4952793b-a587-46ca-bcb5-e9c94e359332-istio-podinfo\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.889034 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.888987 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-envoy\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.889034 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.889037 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4952793b-a587-46ca-bcb5-e9c94e359332-istio-podinfo\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.889289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.889109 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4952793b-a587-46ca-bcb5-e9c94e359332-istiod-ca-cert\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.889289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.889137 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-socket\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.889289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.889196 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-istio-token\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.889289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.889251 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwlm\" (UniqueName: \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-kube-api-access-9nwlm\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.889289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.889278 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-credential-socket\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.889602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.889318 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-data\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.889602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.889494 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-certs\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.889975 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.889952 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-certs\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.891291 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.891265 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-data\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.892088 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.892065 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4952793b-a587-46ca-bcb5-e9c94e359332-istiod-ca-cert\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.892825 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.892300 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-credential-socket\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.892991 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.892974 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-socket\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.896127 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.896105 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4952793b-a587-46ca-bcb5-e9c94e359332-istio-podinfo\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.897562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.897529 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-envoy\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.916969 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.916879 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwlm\" (UniqueName: \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-kube-api-access-9nwlm\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:56.917313 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:56.917298 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-istio-token\") pod \"e2e-explicit-389dba-openshift-default-6875794477-rb6gh\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:57.044102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.044065 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:57.124348 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.121634 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm"] Oct 30 01:47:57.130575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.127636 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.148913 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.148889 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-396f8e\"/\"openshift-service-ca.crt\"" Oct 30 01:47:57.149140 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.149126 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-396f8e\"/\"kube-root-ca.crt\"" Oct 30 01:47:57.194869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.194316 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-dshm\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.194869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.194411 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-home\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.194869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.194443 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx72m\" (UniqueName: \"kubernetes.io/projected/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-kube-api-access-lx72m\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.194869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.194476 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-model-cache\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.194869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.194520 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tls-certs\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.194869 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.194608 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tmp-dir\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.197590 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.197559 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm"] Oct 30 01:47:57.214792 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.213707 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-396f8e\"/\"default-dockercfg-pt59q\"" Oct 30 01:47:57.214792 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.213830 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-396f8e\"/\"auth-test-model-396f8e-kserve-self-signed-certs\"" Oct 30 01:47:57.309374 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.309269 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-dshm\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.309556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.309370 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-home\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.309556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.309406 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx72m\" (UniqueName: \"kubernetes.io/projected/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-kube-api-access-lx72m\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.309556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.309441 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-model-cache\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.309556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.309487 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tls-certs\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.309772 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.309589 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tmp-dir\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.310071 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.310041 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tmp-dir\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.310958 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.310931 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-home\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.311210 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.311186 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-model-cache\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.318908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.318876 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-dshm\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.320907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.320880 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tls-certs\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.329942 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.329913 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx72m\" (UniqueName: \"kubernetes.io/projected/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-kube-api-access-lx72m\") pod \"auth-test-model-396f8e-kserve-7db4f564c7-ms8bm\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.460351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.452676 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:57.500364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.483799 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh"] Oct 30 01:47:57.575417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.574506 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:47:57.575417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.574612 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:47:57.575417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.574678 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:47:57.977233 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:57.977181 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm"] Oct 30 01:47:57.981419 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:47:57.981382 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dbbeb20_ed61_433a_a0f1_11bcf3948b69.slice/crio-8976fba551da8a1b3c0867b58628ae42d773435946b09f565f8ec876354613d9 WatchSource:0}: Error finding container 8976fba551da8a1b3c0867b58628ae42d773435946b09f565f8ec876354613d9: Status 404 returned error can't find the container with id 8976fba551da8a1b3c0867b58628ae42d773435946b09f565f8ec876354613d9 Oct 30 01:47:58.158544 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:58.158510 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" event={"ID":"2dbbeb20-ed61-433a-a0f1-11bcf3948b69","Type":"ContainerStarted","Data":"df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca"} Oct 30 01:47:58.158718 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:58.158554 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" event={"ID":"2dbbeb20-ed61-433a-a0f1-11bcf3948b69","Type":"ContainerStarted","Data":"8976fba551da8a1b3c0867b58628ae42d773435946b09f565f8ec876354613d9"} Oct 30 01:47:58.160702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:58.160670 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" event={"ID":"4952793b-a587-46ca-bcb5-e9c94e359332","Type":"ContainerStarted","Data":"6a8c45fdcaefdc0a3b4ecc0712c6c4fc77462a7477d428b977231bd77227e24b"} Oct 30 01:47:58.160855 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:58.160722 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" event={"ID":"4952793b-a587-46ca-bcb5-e9c94e359332","Type":"ContainerStarted","Data":"5aeb3f69c5143128543c38e767ac5bbf3a8b5bf1486ef1fd5a518e44510ec046"} Oct 30 01:47:58.202346 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:58.201641 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" podStartSLOduration=1.201624048 podStartE2EDuration="1.201624048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:58.200178324 +0000 UTC m=+1845.521135366" watchObservedRunningTime="2026-10-30 01:47:58.201624048 +0000 UTC m=+1845.522581091" Oct 30 01:47:58.244752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:58.244664 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" podStartSLOduration=2.244650829 podStartE2EDuration="2.244650829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:58.243560667 +0000 UTC m=+1845.564517718" watchObservedRunningTime="2026-10-30 01:47:58.244650829 +0000 UTC m=+1845.565607943" Oct 30 01:47:58.459350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:58.458426 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:47:59.044394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:59.044362 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:59.050176 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:59.050155 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:59.167129 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:59.167092 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:47:59.168165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:47:59.168144 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:48:00.452219 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.452189 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:48:00.477799 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.477774 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-socket\") pod \"6ae8f195-ce05-4880-8cc0-a76799258ff0\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " Oct 30 01:48:00.477998 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.477814 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-data\") pod \"6ae8f195-ce05-4880-8cc0-a76799258ff0\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " Oct 30 01:48:00.477998 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.477872 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-certs\") pod \"6ae8f195-ce05-4880-8cc0-a76799258ff0\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " Oct 30 01:48:00.477998 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.477913 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-token\" (UniqueName: \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-token\") pod \"6ae8f195-ce05-4880-8cc0-a76799258ff0\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " Oct 30 01:48:00.478169 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478052 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae8f195-ce05-4880-8cc0-a76799258ff0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae8f195-ce05-4880-8cc0-a76799258ff0-istiod-ca-cert\") pod \"6ae8f195-ce05-4880-8cc0-a76799258ff0\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " Oct 30 01:48:00.478169 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478100 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-socket" pod "6ae8f195-ce05-4880-8cc0-a76799258ff0" (UID: "6ae8f195-ce05-4880-8cc0-a76799258ff0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:00.478169 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478114 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-credential-socket\") pod \"6ae8f195-ce05-4880-8cc0-a76799258ff0\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " Oct 30 01:48:00.478337 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478195 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-podinfo\") pod \"6ae8f195-ce05-4880-8cc0-a76799258ff0\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " Oct 30 01:48:00.478337 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478185 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-data" pod "6ae8f195-ce05-4880-8cc0-a76799258ff0" (UID: "6ae8f195-ce05-4880-8cc0-a76799258ff0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:00.478337 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478239 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-kube-api-access-2t5cz\" (UniqueName: \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-kube-api-access-2t5cz\") pod \"6ae8f195-ce05-4880-8cc0-a76799258ff0\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " Oct 30 01:48:00.478337 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478256 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-certs" pod "6ae8f195-ce05-4880-8cc0-a76799258ff0" (UID: "6ae8f195-ce05-4880-8cc0-a76799258ff0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:00.478555 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478341 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-envoy\") pod \"6ae8f195-ce05-4880-8cc0-a76799258ff0\" (UID: \"6ae8f195-ce05-4880-8cc0-a76799258ff0\") " Oct 30 01:48:00.478555 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478388 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-credential-socket" pod "6ae8f195-ce05-4880-8cc0-a76799258ff0" (UID: "6ae8f195-ce05-4880-8cc0-a76799258ff0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:00.478928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478881 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae8f195-ce05-4880-8cc0-a76799258ff0-istiod-ca-cert" pod "6ae8f195-ce05-4880-8cc0-a76799258ff0" (UID: "6ae8f195-ce05-4880-8cc0-a76799258ff0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:48:00.478928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478889 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:48:00.478928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478930 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:48:00.478928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478947 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:48:00.478928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.478960 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:48:00.480672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.480651 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-kube-api-access-2t5cz" pod "6ae8f195-ce05-4880-8cc0-a76799258ff0" (UID: "6ae8f195-ce05-4880-8cc0-a76799258ff0"). InnerVolumeSpecName "kube-api-access-2t5cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:00.481252 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.481225 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-envoy" pod "6ae8f195-ce05-4880-8cc0-a76799258ff0" (UID: "6ae8f195-ce05-4880-8cc0-a76799258ff0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:48:00.481252 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.481229 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-podinfo" pod "6ae8f195-ce05-4880-8cc0-a76799258ff0" (UID: "6ae8f195-ce05-4880-8cc0-a76799258ff0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:48:00.482701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.482679 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-token" pod "6ae8f195-ce05-4880-8cc0-a76799258ff0" (UID: "6ae8f195-ce05-4880-8cc0-a76799258ff0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:48:00.580440 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.580409 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ae8f195-ce05-4880-8cc0-a76799258ff0-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:48:00.580440 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.580431 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:48:00.580440 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.580441 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t5cz\" (UniqueName: \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-kube-api-access-2t5cz\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:48:00.580728 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.580453 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:48:00.580728 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:00.580461 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ae8f195-ce05-4880-8cc0-a76799258ff0-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:48:01.183130 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.183041 5055 generic.go:356] "Generic (PLEG): container finished" podID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerID="9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e" exitCode=0 Oct 30 01:48:01.183130 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.183093 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" Oct 30 01:48:01.183377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.183126 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" event={"ID":"6ae8f195-ce05-4880-8cc0-a76799258ff0","Type":"ContainerDied","Data":"9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e"} Oct 30 01:48:01.183377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.183203 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr" event={"ID":"6ae8f195-ce05-4880-8cc0-a76799258ff0","Type":"ContainerDied","Data":"5151dfac0bf01489d20d9fd84fd6a362d22dce78db543f8785d4e164104f9db8"} Oct 30 01:48:01.183377 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.183284 5055 scope.go:122] "RemoveContainer" containerID="9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e" Oct 30 01:48:01.195287 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.195243 5055 scope.go:122] "RemoveContainer" containerID="561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8" Oct 30 01:48:01.208435 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.208410 5055 scope.go:122] "RemoveContainer" containerID="9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e" Oct 30 01:48:01.208844 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:48:01.208823 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e\": container with ID starting with 9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e not found: ID does not exist" containerID="9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e" Oct 30 01:48:01.208939 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.208857 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e"} err="failed to get container status \"9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e\": rpc error: code = NotFound desc = could not find container \"9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e\": container with ID starting with 9334a8ff0179f8b90550807a40b7503ce985600562177d9515cd9331abd4644e not found: ID does not exist" Oct 30 01:48:01.208939 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.208901 5055 scope.go:122] "RemoveContainer" containerID="561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8" Oct 30 01:48:01.209240 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:48:01.209215 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8\": container with ID starting with 561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8 not found: ID does not exist" containerID="561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8" Oct 30 01:48:01.209296 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.209250 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8"} err="failed to get container status \"561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8\": rpc error: code = NotFound desc = could not find container \"561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8\": container with ID starting with 561681daf7c501f0829b56365d316b3ade00446dba41e455e81e166b73324ac8 not found: ID does not exist" Oct 30 01:48:01.249965 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.249926 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr"] Oct 30 01:48:01.256584 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:01.256551 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-58632b-openshift-default-69df587c49-w5sdr"] Oct 30 01:48:02.832993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:02.832957 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" path="/var/lib/kubelet/pods/6ae8f195-ce05-4880-8cc0-a76799258ff0/volumes" Oct 30 01:48:03.237127 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.237090 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn"] Oct 30 01:48:03.238384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.238358 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" Oct 30 01:48:03.239981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.239958 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" Oct 30 01:48:03.240139 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.240127 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" Oct 30 01:48:03.240227 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.240216 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" Oct 30 01:48:03.241975 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.241954 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" Oct 30 01:48:03.242112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.242100 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae8f195-ce05-4880-8cc0-a76799258ff0" containerName="istio-proxy" Oct 30 01:48:03.249166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.249140 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn"] Oct 30 01:48:03.249394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.249245 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.269906 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.269857 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-861cd2-serving-cert\"" Oct 30 01:48:03.367896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.367862 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.368076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.367974 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.368076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.368045 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdxxt\" (UniqueName: \"kubernetes.io/projected/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-kube-api-access-jdxxt\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.368159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.368081 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.469416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.469378 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.469640 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.469525 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdxxt\" (UniqueName: \"kubernetes.io/projected/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-kube-api-access-jdxxt\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.470055 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.470031 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.470215 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.470195 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.470771 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.470753 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.473919 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.473895 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.475839 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.475811 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.491498 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.491390 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdxxt\" (UniqueName: \"kubernetes.io/projected/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-kube-api-access-jdxxt\") pod \"maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.574180 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.574140 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:03.884201 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:03.884177 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn"] Oct 30 01:48:03.884585 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:48:03.884172 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd208ed75_9ca5_4dfc_9931_0fc4e5276d0e.slice/crio-1aa0c302d084678cd73ba03265b777d19ee4b95693342d1ee5938990094ae0bc WatchSource:0}: Error finding container 1aa0c302d084678cd73ba03265b777d19ee4b95693342d1ee5938990094ae0bc: Status 404 returned error can't find the container with id 1aa0c302d084678cd73ba03265b777d19ee4b95693342d1ee5938990094ae0bc Oct 30 01:48:04.244428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:04.244389 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" event={"ID":"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e","Type":"ContainerStarted","Data":"1aa0c302d084678cd73ba03265b777d19ee4b95693342d1ee5938990094ae0bc"} Oct 30 01:48:05.252553 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:05.252518 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" event={"ID":"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e","Type":"ContainerStarted","Data":"9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19"} Oct 30 01:48:05.252964 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:05.252720 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:05.292827 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:05.292766 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" podStartSLOduration=1.913557934 podStartE2EDuration="2.292746221s" totalImagesPullingTime="379.188287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:05.290898351 +0000 UTC m=+1852.611855390" watchObservedRunningTime="2026-10-30 01:48:05.292746221 +0000 UTC m=+1852.613703263" Oct 30 01:48:07.457015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:07.456977 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:48:07.470077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:07.470050 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:48:08.287666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:08.287637 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:48:11.267251 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:11.267220 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 01:48:17.668881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.668843 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw"] Oct 30 01:48:17.674477 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.674456 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.698680 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.698652 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw"] Oct 30 01:48:17.704055 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.704014 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-264bdd-serving-cert\"" Oct 30 01:48:17.792395 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.792317 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.792579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.792533 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.792663 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.792577 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.792726 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.792698 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxpz6\" (UniqueName: \"kubernetes.io/projected/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-kube-api-access-pxpz6\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.893820 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.893783 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.894029 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.893850 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.894029 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.893918 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxpz6\" (UniqueName: \"kubernetes.io/projected/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-kube-api-access-pxpz6\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.894165 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.894135 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.895079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.895045 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.897469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.897437 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.897833 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.897811 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.913901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.913840 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxpz6\" (UniqueName: \"kubernetes.io/projected/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-kube-api-access-pxpz6\") pod \"maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:17.989732 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:17.989461 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:18.311282 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:48:18.311253 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d58e2b_8457_4d5a_b096_dd5ceb25ca6b.slice/crio-f00c0b14b57431b3d1a0a93bdf59f35dd0cd8c41c7b38d052ad109017c56dc9f WatchSource:0}: Error finding container f00c0b14b57431b3d1a0a93bdf59f35dd0cd8c41c7b38d052ad109017c56dc9f: Status 404 returned error can't find the container with id f00c0b14b57431b3d1a0a93bdf59f35dd0cd8c41c7b38d052ad109017c56dc9f Oct 30 01:48:18.313119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:18.313097 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw"] Oct 30 01:48:18.389075 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:18.389032 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" event={"ID":"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b","Type":"ContainerStarted","Data":"f00c0b14b57431b3d1a0a93bdf59f35dd0cd8c41c7b38d052ad109017c56dc9f"} Oct 30 01:48:19.227479 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:48:19.227441 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:48:19.398635 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:19.398599 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" event={"ID":"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b","Type":"ContainerStarted","Data":"b933bf327dfe28ad1186dc0d797c44b095342cd22ae670650065e53094d288cb"} Oct 30 01:48:19.398812 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:19.398671 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:19.438822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:19.438768 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" podStartSLOduration=2.01318604 podStartE2EDuration="2.438753387s" totalImagesPullingTime="425.567347ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:19.437755248 +0000 UTC m=+1866.758712302" watchObservedRunningTime="2026-10-30 01:48:19.438753387 +0000 UTC m=+1866.759710427" Oct 30 01:48:22.806955 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.806915 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4"] Oct 30 01:48:22.818524 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.818489 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.836338 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.836291 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4"] Oct 30 01:48:22.838745 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.838711 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-maas-api-tls\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.838888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.838767 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-metrics-certs\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.838975 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.838944 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-openshift-service-ca\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.839038 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.838996 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x8jg\" (UniqueName: \"kubernetes.io/projected/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-kube-api-access-5x8jg\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.845421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.841979 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-389dba-serving-cert\"" Oct 30 01:48:22.939994 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.939960 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-openshift-service-ca\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.939994 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.940009 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x8jg\" (UniqueName: \"kubernetes.io/projected/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-kube-api-access-5x8jg\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.940491 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.940466 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-maas-api-tls\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.940601 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.940519 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-metrics-certs\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.940662 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.940614 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-openshift-service-ca\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.940774 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:48:22.940759 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-serving-cert: secret "maas-api-e2e-explicit-389dba-serving-cert" not found Oct 30 01:48:22.940842 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:48:22.940831 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-maas-api-tls podName:ff96e34c-28ce-4bcc-bd42-7e52dec45c7d nodeName:}" failed. No retries permitted until 2026-10-30 01:48:23.440814402 +0000 UTC m=+1870.761771437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-maas-api-tls") pod "maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" (UID: "ff96e34c-28ce-4bcc-bd42-7e52dec45c7d") : secret "maas-api-e2e-explicit-389dba-serving-cert" not found Oct 30 01:48:22.943345 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.943295 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-metrics-certs\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:22.963190 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:22.963132 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x8jg\" (UniqueName: \"kubernetes.io/projected/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-kube-api-access-5x8jg\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:23.448837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:23.448804 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-maas-api-tls\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:23.451479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:23.451453 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-maas-api-tls\") pod \"maas-api-e2e-explicit-389dba-67cfd756d6-b94s4\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:23.746626 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:23.746589 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:24.121182 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:24.116745 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4"] Oct 30 01:48:24.439432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:24.439307 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" event={"ID":"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d","Type":"ContainerStarted","Data":"7e81af7bf5438f75a094d48990d7066fb673b4e2b47f3d12e52270549b91853b"} Oct 30 01:48:25.415262 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:25.415230 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 01:48:25.448275 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:25.448244 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" event={"ID":"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d","Type":"ContainerStarted","Data":"d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785"} Oct 30 01:48:25.448461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:25.448309 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:31.460815 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:31.460782 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:48:31.514450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:31.514283 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" podStartSLOduration=9.101365261 podStartE2EDuration="9.514243631s" totalImagesPullingTime="412.87837ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:25.515635738 +0000 UTC m=+1872.836592805" watchObservedRunningTime="2026-10-30 01:48:31.514243631 +0000 UTC m=+1878.835200673" Oct 30 01:48:46.267991 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.267943 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v"] Oct 30 01:48:46.273225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.273197 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.327815 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.327238 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-abb30f\"/\"auth-test-model-abb30f-kserve-self-signed-certs\"" Oct 30 01:48:46.329016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.328976 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v"] Oct 30 01:48:46.329184 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.329168 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-abb30f\"/\"openshift-service-ca.crt\"" Oct 30 01:48:46.330882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.330194 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-abb30f\"/\"default-dockercfg-mf6qd\"" Oct 30 01:48:46.330882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.330275 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-abb30f\"/\"kube-root-ca.crt\"" Oct 30 01:48:46.363908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.363195 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-tmp-dir\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.363908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.363280 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-model-cache\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.363908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.363309 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-home\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.363908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.363361 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96fvj\" (UniqueName: \"kubernetes.io/projected/72fa1a14-4f43-43db-99eb-55f0c55f2924-kube-api-access-96fvj\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.363908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.363413 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72fa1a14-4f43-43db-99eb-55f0c55f2924-tls-certs\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.363908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.363476 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-dshm\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.466234 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.466191 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-tmp-dir\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.466479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.466278 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-model-cache\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.466479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.466310 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-home\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.466479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.466359 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96fvj\" (UniqueName: \"kubernetes.io/projected/72fa1a14-4f43-43db-99eb-55f0c55f2924-kube-api-access-96fvj\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.466479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.466414 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72fa1a14-4f43-43db-99eb-55f0c55f2924-tls-certs\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.466479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.466474 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-dshm\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.466763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.466715 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-tmp-dir\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.466763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.466750 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-model-cache\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.466950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.466923 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-home\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.469800 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.469775 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-dshm\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.470146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.470120 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72fa1a14-4f43-43db-99eb-55f0c55f2924-tls-certs\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.488762 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.488731 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96fvj\" (UniqueName: \"kubernetes.io/projected/72fa1a14-4f43-43db-99eb-55f0c55f2924-kube-api-access-96fvj\") pod \"auth-test-model-abb30f-kserve-7b77d5db9-tff6v\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.636981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.635708 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:46.997316 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:48:46.997282 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72fa1a14_4f43_43db_99eb_55f0c55f2924.slice/crio-ad44bc0e42ce6daa0a8c1971d8d61049f39f1b928ffc402fbb67556a438b8fa2 WatchSource:0}: Error finding container ad44bc0e42ce6daa0a8c1971d8d61049f39f1b928ffc402fbb67556a438b8fa2: Status 404 returned error can't find the container with id ad44bc0e42ce6daa0a8c1971d8d61049f39f1b928ffc402fbb67556a438b8fa2 Oct 30 01:48:46.999028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:46.998998 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v"] Oct 30 01:48:47.660126 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:47.660082 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" event={"ID":"72fa1a14-4f43-43db-99eb-55f0c55f2924","Type":"ContainerStarted","Data":"d128ae2709083489239b1b375fc7f1963feb467b69e2aba8ee28b492b303df07"} Oct 30 01:48:47.660126 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:47.660129 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" event={"ID":"72fa1a14-4f43-43db-99eb-55f0c55f2924","Type":"ContainerStarted","Data":"ad44bc0e42ce6daa0a8c1971d8d61049f39f1b928ffc402fbb67556a438b8fa2"} Oct 30 01:48:47.709658 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:47.709604 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" podStartSLOduration=1.709584378 podStartE2EDuration="1.709584378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:47.705877754 +0000 UTC m=+1895.026834794" watchObservedRunningTime="2026-10-30 01:48:47.709584378 +0000 UTC m=+1895.030541503" Oct 30 01:48:48.637462 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:48.637428 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:49.222695 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:48:49.222660 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:48:50.686833 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:50.686804 5055 generic.go:356] "Generic (PLEG): container finished" podID="4952793b-a587-46ca-bcb5-e9c94e359332" containerID="6a8c45fdcaefdc0a3b4ecc0712c6c4fc77462a7477d428b977231bd77227e24b" exitCode=0 Oct 30 01:48:50.687318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:50.686848 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" event={"ID":"4952793b-a587-46ca-bcb5-e9c94e359332","Type":"ContainerDied","Data":"6a8c45fdcaefdc0a3b4ecc0712c6c4fc77462a7477d428b977231bd77227e24b"} Oct 30 01:48:50.687318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:50.687284 5055 scope.go:122] "RemoveContainer" containerID="6a8c45fdcaefdc0a3b4ecc0712c6c4fc77462a7477d428b977231bd77227e24b" Oct 30 01:48:50.689016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:50.688988 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:48:50.689090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:50.689059 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:48:50.689131 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:50.689092 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:48:51.707652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:51.707612 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" event={"ID":"4952793b-a587-46ca-bcb5-e9c94e359332","Type":"ContainerStarted","Data":"ed65beb62963ad9eeefcfb4566fdbb95a2fb19ea284b9f045fcda5edfbaaa4f7"} Oct 30 01:48:52.044902 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:52.044827 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:48:52.045046 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:52.044989 5055 patch_prober.go:29] interesting pod/e2e-explicit-389dba-openshift-default-6875794477-rb6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:48:52.045100 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:52.045034 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 01:48:53.045114 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:53.045082 5055 patch_prober.go:29] interesting pod/e2e-explicit-389dba-openshift-default-6875794477-rb6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:48:53.045587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:53.045150 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 01:48:54.044623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:54.044587 5055 patch_prober.go:29] interesting pod/e2e-explicit-389dba-openshift-default-6875794477-rb6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:48:54.044809 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:54.044651 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 01:48:55.045098 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:55.045058 5055 patch_prober.go:29] interesting pod/e2e-explicit-389dba-openshift-default-6875794477-rb6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:48:55.045578 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:55.045140 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 01:48:56.045221 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:56.045174 5055 patch_prober.go:29] interesting pod/e2e-explicit-389dba-openshift-default-6875794477-rb6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:48:56.045708 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:56.045230 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 01:48:56.637531 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:56.637495 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:56.650556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:56.650525 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:56.826558 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:56.826528 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:48:57.044942 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:57.044909 5055 patch_prober.go:29] interesting pod/e2e-explicit-389dba-openshift-default-6875794477-rb6gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Oct 30 01:48:57.045144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:57.044972 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Oct 30 01:48:58.050163 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:58.050129 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:48:58.050621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:58.050599 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:48:58.051540 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:48:58.051517 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:49:19.139572 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.139538 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n"] Oct 30 01:49:19.145110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.145085 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.171421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.171382 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n"] Oct 30 01:49:19.180653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.180625 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-389dba\"/\"kube-root-ca.crt\"" Oct 30 01:49:19.180822 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.180684 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-389dba\"/\"openshift-service-ca.crt\"" Oct 30 01:49:19.180906 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.180831 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-389dba\"/\"explicit-model-389dba-kserve-self-signed-certs\"" Oct 30 01:49:19.181001 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.180980 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-389dba\"/\"default-dockercfg-pd58v\"" Oct 30 01:49:19.238092 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:49:19.238056 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:49:19.243145 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.243114 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-dshm\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.243314 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.243258 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e18e013e-b517-4957-8f8e-2b74d5b996ae-tls-certs\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.243314 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.243300 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-home\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.243457 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.243400 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-tmp-dir\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.243658 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.243635 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsnrd\" (UniqueName: \"kubernetes.io/projected/e18e013e-b517-4957-8f8e-2b74d5b996ae-kube-api-access-lsnrd\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.243730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.243688 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-model-cache\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.344810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.344771 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsnrd\" (UniqueName: \"kubernetes.io/projected/e18e013e-b517-4957-8f8e-2b74d5b996ae-kube-api-access-lsnrd\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.345022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.344821 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-model-cache\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.345022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.344962 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-dshm\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.345402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.345375 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-model-cache\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.345495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.345460 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e18e013e-b517-4957-8f8e-2b74d5b996ae-tls-certs\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.345495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.345487 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-home\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.345617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.345543 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-tmp-dir\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.345967 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.345941 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-tmp-dir\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.346123 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.346086 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-home\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.347718 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.347690 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-dshm\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.348421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.348397 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e18e013e-b517-4957-8f8e-2b74d5b996ae-tls-certs\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.367008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.366975 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsnrd\" (UniqueName: \"kubernetes.io/projected/e18e013e-b517-4957-8f8e-2b74d5b996ae-kube-api-access-lsnrd\") pod \"explicit-model-389dba-kserve-57f497cbcc-kwb8n\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.506540 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.506492 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:19.856882 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:49:19.856844 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode18e013e_b517_4957_8f8e_2b74d5b996ae.slice/crio-efbe2e4307ba331381f2434428e9d19db1ff247d9ed524cba8b6086488e27dfa WatchSource:0}: Error finding container efbe2e4307ba331381f2434428e9d19db1ff247d9ed524cba8b6086488e27dfa: Status 404 returned error can't find the container with id efbe2e4307ba331381f2434428e9d19db1ff247d9ed524cba8b6086488e27dfa Oct 30 01:49:19.858005 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:19.857978 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n"] Oct 30 01:49:20.009804 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:20.009765 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" event={"ID":"e18e013e-b517-4957-8f8e-2b74d5b996ae","Type":"ContainerStarted","Data":"b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d"} Oct 30 01:49:20.010001 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:20.009818 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" event={"ID":"e18e013e-b517-4957-8f8e-2b74d5b996ae","Type":"ContainerStarted","Data":"efbe2e4307ba331381f2434428e9d19db1ff247d9ed524cba8b6086488e27dfa"} Oct 30 01:49:20.052364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:20.051320 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" podStartSLOduration=1.051300704 podStartE2EDuration="1.051300704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:20.0487168 +0000 UTC m=+1927.369673841" watchObservedRunningTime="2026-10-30 01:49:20.051300704 +0000 UTC m=+1927.372257745" Oct 30 01:49:20.508976 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:20.508938 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:29.508846 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:29.508750 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:29.523484 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:29.523449 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:30.096510 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:30.096480 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:35.835367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:35.835304 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n"] Oct 30 01:49:49.223090 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:49:49.223055 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:49:50.865269 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:50.865232 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" podUID="e18e013e-b517-4957-8f8e-2b74d5b996ae" containerName="main" containerID="cri-o://b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d" gracePeriod=45 Oct 30 01:49:51.215845 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.215815 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:51.310796 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.310763 5055 generic.go:356] "Generic (PLEG): container finished" podID="e18e013e-b517-4957-8f8e-2b74d5b996ae" containerID="b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d" exitCode=0 Oct 30 01:49:51.310995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.310821 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" Oct 30 01:49:51.310995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.310846 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" event={"ID":"e18e013e-b517-4957-8f8e-2b74d5b996ae","Type":"ContainerDied","Data":"b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d"} Oct 30 01:49:51.310995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.310885 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" event={"ID":"e18e013e-b517-4957-8f8e-2b74d5b996ae","Type":"ContainerDied","Data":"efbe2e4307ba331381f2434428e9d19db1ff247d9ed524cba8b6086488e27dfa"} Oct 30 01:49:51.310995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.310906 5055 scope.go:122] "RemoveContainer" containerID="b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d" Oct 30 01:49:51.311554 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.311527 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-dshm\") pod \"e18e013e-b517-4957-8f8e-2b74d5b996ae\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " Oct 30 01:49:51.311715 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.311568 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e18e013e-b517-4957-8f8e-2b74d5b996ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/e18e013e-b517-4957-8f8e-2b74d5b996ae-tls-certs\") pod \"e18e013e-b517-4957-8f8e-2b74d5b996ae\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " Oct 30 01:49:51.311715 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.311648 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e18e013e-b517-4957-8f8e-2b74d5b996ae-kube-api-access-lsnrd\" (UniqueName: \"kubernetes.io/projected/e18e013e-b517-4957-8f8e-2b74d5b996ae-kube-api-access-lsnrd\") pod \"e18e013e-b517-4957-8f8e-2b74d5b996ae\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " Oct 30 01:49:51.311715 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.311696 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-tmp-dir\") pod \"e18e013e-b517-4957-8f8e-2b74d5b996ae\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " Oct 30 01:49:51.311906 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.311774 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-model-cache\") pod \"e18e013e-b517-4957-8f8e-2b74d5b996ae\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " Oct 30 01:49:51.311971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.311953 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-home\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-home\") pod \"e18e013e-b517-4957-8f8e-2b74d5b996ae\" (UID: \"e18e013e-b517-4957-8f8e-2b74d5b996ae\") " Oct 30 01:49:51.312156 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.312134 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-tmp-dir" pod "e18e013e-b517-4957-8f8e-2b74d5b996ae" (UID: "e18e013e-b517-4957-8f8e-2b74d5b996ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:51.312449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.312418 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-home" pod "e18e013e-b517-4957-8f8e-2b74d5b996ae" (UID: "e18e013e-b517-4957-8f8e-2b74d5b996ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:51.312745 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.312726 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:49:51.312840 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.312751 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:49:51.312840 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.312813 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-model-cache" pod "e18e013e-b517-4957-8f8e-2b74d5b996ae" (UID: "e18e013e-b517-4957-8f8e-2b74d5b996ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:51.315533 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.315509 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e18e013e-b517-4957-8f8e-2b74d5b996ae-kube-api-access-lsnrd" pod "e18e013e-b517-4957-8f8e-2b74d5b996ae" (UID: "e18e013e-b517-4957-8f8e-2b74d5b996ae"). InnerVolumeSpecName "kube-api-access-lsnrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:51.316769 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.316673 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e18e013e-b517-4957-8f8e-2b74d5b996ae-tls-certs" pod "e18e013e-b517-4957-8f8e-2b74d5b996ae" (UID: "e18e013e-b517-4957-8f8e-2b74d5b996ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:49:51.316769 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.316737 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-dshm" pod "e18e013e-b517-4957-8f8e-2b74d5b996ae" (UID: "e18e013e-b517-4957-8f8e-2b74d5b996ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:49:51.345398 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.345364 5055 scope.go:122] "RemoveContainer" containerID="b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d" Oct 30 01:49:51.345756 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:49:51.345737 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d\": container with ID starting with b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d not found: ID does not exist" containerID="b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d" Oct 30 01:49:51.345804 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.345773 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d"} err="failed to get container status \"b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d\": rpc error: code = NotFound desc = could not find container \"b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d\": container with ID starting with b6b99df2476c50407c74409b7f51657fdfa32edc5b106229bd7add186d14a35d not found: ID does not exist" Oct 30 01:49:51.414112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.414078 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:49:51.414112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.414110 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e18e013e-b517-4957-8f8e-2b74d5b996ae-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:49:51.414112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.414122 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsnrd\" (UniqueName: \"kubernetes.io/projected/e18e013e-b517-4957-8f8e-2b74d5b996ae-kube-api-access-lsnrd\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:49:51.414409 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.414132 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e18e013e-b517-4957-8f8e-2b74d5b996ae-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:49:51.658442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.658413 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n"] Oct 30 01:49:51.672830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:51.672791 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n"] Oct 30 01:49:52.084133 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:52.084089 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-explicit-389dba/explicit-model-389dba-kserve-57f497cbcc-kwb8n" podUID="e18e013e-b517-4957-8f8e-2b74d5b996ae" containerName="main" probeResult="failure" output="Get \"https://10.132.0.148:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:49:52.822589 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:49:52.822562 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e18e013e-b517-4957-8f8e-2b74d5b996ae" path="/var/lib/kubelet/pods/e18e013e-b517-4957-8f8e-2b74d5b996ae/volumes" Oct 30 01:50:07.331360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:07.329667 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4"] Oct 30 01:50:07.331360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:07.329928 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" podUID="ff96e34c-28ce-4bcc-bd42-7e52dec45c7d" containerName="maas-api" containerID="cri-o://d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785" gracePeriod=30 Oct 30 01:50:07.927701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:07.906878 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:50:07.927701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:07.915143 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-metrics-certs\") pod \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " Oct 30 01:50:07.927701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:07.915213 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-kube-api-access-5x8jg\" (UniqueName: \"kubernetes.io/projected/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-kube-api-access-5x8jg\") pod \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " Oct 30 01:50:07.927701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:07.915296 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-maas-api-tls\") pod \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " Oct 30 01:50:07.927701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:07.915406 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-openshift-service-ca\") pod \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\" (UID: \"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d\") " Oct 30 01:50:07.959350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:07.956796 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-openshift-service-ca" pod "ff96e34c-28ce-4bcc-bd42-7e52dec45c7d" (UID: "ff96e34c-28ce-4bcc-bd42-7e52dec45c7d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:50:08.034587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.026681 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-kube-api-access-5x8jg" pod "ff96e34c-28ce-4bcc-bd42-7e52dec45c7d" (UID: "ff96e34c-28ce-4bcc-bd42-7e52dec45c7d"). InnerVolumeSpecName "kube-api-access-5x8jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:50:08.034587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.030174 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x8jg\" (UniqueName: \"kubernetes.io/projected/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-kube-api-access-5x8jg\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:08.034587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.030206 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:08.045981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.036491 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-maas-api-tls" pod "ff96e34c-28ce-4bcc-bd42-7e52dec45c7d" (UID: "ff96e34c-28ce-4bcc-bd42-7e52dec45c7d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:50:08.045981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.036584 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-metrics-certs" pod "ff96e34c-28ce-4bcc-bd42-7e52dec45c7d" (UID: "ff96e34c-28ce-4bcc-bd42-7e52dec45c7d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:50:08.132229 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.132192 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:08.132501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.132487 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:08.459297 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.459265 5055 generic.go:356] "Generic (PLEG): container finished" podID="ff96e34c-28ce-4bcc-bd42-7e52dec45c7d" containerID="d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785" exitCode=0 Oct 30 01:50:08.459748 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.459375 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" event={"ID":"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d","Type":"ContainerDied","Data":"d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785"} Oct 30 01:50:08.459748 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.459388 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" Oct 30 01:50:08.459748 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.459430 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4" event={"ID":"ff96e34c-28ce-4bcc-bd42-7e52dec45c7d","Type":"ContainerDied","Data":"7e81af7bf5438f75a094d48990d7066fb673b4e2b47f3d12e52270549b91853b"} Oct 30 01:50:08.459748 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.459463 5055 scope.go:122] "RemoveContainer" containerID="d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785" Oct 30 01:50:08.490416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.490369 5055 scope.go:122] "RemoveContainer" containerID="d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785" Oct 30 01:50:08.490846 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:50:08.490819 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785\": container with ID starting with d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785 not found: ID does not exist" containerID="d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785" Oct 30 01:50:08.490971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.490859 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785"} err="failed to get container status \"d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785\": rpc error: code = NotFound desc = could not find container \"d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785\": container with ID starting with d694a342f74c7947ce892dd380bd1acb25c75e0bd8ef015217d6b9a6ea8f9785 not found: ID does not exist" Oct 30 01:50:08.551826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.551781 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4"] Oct 30 01:50:08.564236 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.564200 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-389dba-67cfd756d6-b94s4"] Oct 30 01:50:08.823574 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:08.823538 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff96e34c-28ce-4bcc-bd42-7e52dec45c7d" path="/var/lib/kubelet/pods/ff96e34c-28ce-4bcc-bd42-7e52dec45c7d/volumes" Oct 30 01:50:11.173365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.167465 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8"] Oct 30 01:50:11.173365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.168487 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e18e013e-b517-4957-8f8e-2b74d5b996ae" containerName="main" Oct 30 01:50:11.173365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.168510 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e18e013e-b517-4957-8f8e-2b74d5b996ae" containerName="main" Oct 30 01:50:11.173365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.168533 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ff96e34c-28ce-4bcc-bd42-7e52dec45c7d" containerName="maas-api" Oct 30 01:50:11.173365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.168542 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff96e34c-28ce-4bcc-bd42-7e52dec45c7d" containerName="maas-api" Oct 30 01:50:11.173365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.168735 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e18e013e-b517-4957-8f8e-2b74d5b996ae" containerName="main" Oct 30 01:50:11.173365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.168748 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff96e34c-28ce-4bcc-bd42-7e52dec45c7d" containerName="maas-api" Oct 30 01:50:11.174478 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.174450 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.192500 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.192470 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8"] Oct 30 01:50:11.208692 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.208662 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-657fba\"/\"kube-root-ca.crt\"" Oct 30 01:50:11.208875 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.208720 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-657fba\"/\"default-dockercfg-z5h8r\"" Oct 30 01:50:11.208875 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.208789 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-657fba\"/\"auth-test-model-657fba-kserve-self-signed-certs\"" Oct 30 01:50:11.209453 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.209433 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-657fba\"/\"openshift-service-ca.crt\"" Oct 30 01:50:11.295140 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.295096 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zp7v\" (UniqueName: \"kubernetes.io/projected/deb6acec-32cb-4e83-b9a0-298dc297374c-kube-api-access-9zp7v\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.295395 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.295255 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-home\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.295395 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.295316 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-dshm\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.295540 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.295472 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-tmp-dir\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.295596 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.295548 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb6acec-32cb-4e83-b9a0-298dc297374c-tls-certs\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.295653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.295638 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-model-cache\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.397351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.396967 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-dshm\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.397351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.397072 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-tmp-dir\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.397351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.397142 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb6acec-32cb-4e83-b9a0-298dc297374c-tls-certs\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.397351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.397230 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-model-cache\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.397753 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.397363 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp7v\" (UniqueName: \"kubernetes.io/projected/deb6acec-32cb-4e83-b9a0-298dc297374c-kube-api-access-9zp7v\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.397753 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.397433 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-home\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.401351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.397928 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-home\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.401351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.398854 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-tmp-dir\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.401351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.399101 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-model-cache\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.425396 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.419899 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb6acec-32cb-4e83-b9a0-298dc297374c-tls-certs\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.425396 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.422488 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-dshm\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.425396 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.425245 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zp7v\" (UniqueName: \"kubernetes.io/projected/deb6acec-32cb-4e83-b9a0-298dc297374c-kube-api-access-9zp7v\") pod \"auth-test-model-657fba-kserve-5cc8d4d65c-skpj8\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.511354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.510665 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:11.871126 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:11.870949 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8"] Oct 30 01:50:11.874766 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:50:11.874727 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddeb6acec_32cb_4e83_b9a0_298dc297374c.slice/crio-432fab01875e376e44a490a4702b68591eb0d9c0d26e2d53166e05137feaa089 WatchSource:0}: Error finding container 432fab01875e376e44a490a4702b68591eb0d9c0d26e2d53166e05137feaa089: Status 404 returned error can't find the container with id 432fab01875e376e44a490a4702b68591eb0d9c0d26e2d53166e05137feaa089 Oct 30 01:50:12.523450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:12.523409 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" event={"ID":"deb6acec-32cb-4e83-b9a0-298dc297374c","Type":"ContainerStarted","Data":"0b855d97340715605a5c1b38f307ad619f8bedf9753fbc4408e6e634ff00446d"} Oct 30 01:50:12.523450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:12.523456 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" event={"ID":"deb6acec-32cb-4e83-b9a0-298dc297374c","Type":"ContainerStarted","Data":"432fab01875e376e44a490a4702b68591eb0d9c0d26e2d53166e05137feaa089"} Oct 30 01:50:12.570702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:12.570647 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" podStartSLOduration=1.570633243 podStartE2EDuration="1.570633243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:12.569534591 +0000 UTC m=+1979.890491671" watchObservedRunningTime="2026-10-30 01:50:12.570633243 +0000 UTC m=+1979.891590283" Oct 30 01:50:13.511856 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:13.511821 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:15.421362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:15.418759 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh"] Oct 30 01:50:15.421362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:15.419060 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" containerID="cri-o://ed65beb62963ad9eeefcfb4566fdbb95a2fb19ea284b9f045fcda5edfbaaa4f7" gracePeriod=30 Oct 30 01:50:19.229416 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:50:19.229373 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:50:20.582611 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.582571 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc"] Oct 30 01:50:20.587981 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.587956 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.627574 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.627539 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-36c949\"/\"openshift-service-ca.crt\"" Oct 30 01:50:20.627742 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.627634 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-36c949\"/\"kube-root-ca.crt\"" Oct 30 01:50:20.627742 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.627547 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-36c949\"/\"default-dockercfg-mzf6k\"" Oct 30 01:50:20.627742 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.627740 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-36c949\"/\"noten-model-36c949-kserve-self-signed-certs\"" Oct 30 01:50:20.633289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.633256 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc"] Oct 30 01:50:20.669962 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.669901 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-home\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.670169 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.670041 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qf6l\" (UniqueName: \"kubernetes.io/projected/5ff2a708-3621-4666-9789-c1e99acf4883-kube-api-access-9qf6l\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.670169 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.670124 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-model-cache\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.670318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.670206 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-tmp-dir\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.670318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.670237 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-dshm\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.670318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.670306 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff2a708-3621-4666-9789-c1e99acf4883-tls-certs\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.747810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.747775 5055 generic.go:356] "Generic (PLEG): container finished" podID="4952793b-a587-46ca-bcb5-e9c94e359332" containerID="ed65beb62963ad9eeefcfb4566fdbb95a2fb19ea284b9f045fcda5edfbaaa4f7" exitCode=0 Oct 30 01:50:20.748159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.748098 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" event={"ID":"4952793b-a587-46ca-bcb5-e9c94e359332","Type":"ContainerDied","Data":"ed65beb62963ad9eeefcfb4566fdbb95a2fb19ea284b9f045fcda5edfbaaa4f7"} Oct 30 01:50:20.748371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.748358 5055 scope.go:122] "RemoveContainer" containerID="6a8c45fdcaefdc0a3b4ecc0712c6c4fc77462a7477d428b977231bd77227e24b" Oct 30 01:50:20.772144 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.772100 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-home\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.772299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.772224 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qf6l\" (UniqueName: \"kubernetes.io/projected/5ff2a708-3621-4666-9789-c1e99acf4883-kube-api-access-9qf6l\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.772385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.772301 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-model-cache\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.772444 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.772381 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-tmp-dir\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.772444 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.772406 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-dshm\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.772560 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.772464 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff2a708-3621-4666-9789-c1e99acf4883-tls-certs\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.773458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.773431 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-model-cache\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.773599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.773439 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-tmp-dir\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.773699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.773677 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-home\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.776184 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.776159 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-dshm\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.777742 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.777719 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff2a708-3621-4666-9789-c1e99acf4883-tls-certs\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.793795 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.793759 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qf6l\" (UniqueName: \"kubernetes.io/projected/5ff2a708-3621-4666-9789-c1e99acf4883-kube-api-access-9qf6l\") pod \"noten-model-36c949-kserve-5b977d6597-4c5wc\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.823570 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.823534 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:50:20.925351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.924017 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:20.991912 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.991868 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4952793b-a587-46ca-bcb5-e9c94e359332-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4952793b-a587-46ca-bcb5-e9c94e359332-istiod-ca-cert\") pod \"4952793b-a587-46ca-bcb5-e9c94e359332\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " Oct 30 01:50:20.992117 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.991927 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-kube-api-access-9nwlm\" (UniqueName: \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-kube-api-access-9nwlm\") pod \"4952793b-a587-46ca-bcb5-e9c94e359332\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " Oct 30 01:50:20.992117 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.992051 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-data\") pod \"4952793b-a587-46ca-bcb5-e9c94e359332\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " Oct 30 01:50:20.992117 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.992102 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-socket\") pod \"4952793b-a587-46ca-bcb5-e9c94e359332\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " Oct 30 01:50:20.992308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.992147 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4952793b-a587-46ca-bcb5-e9c94e359332-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4952793b-a587-46ca-bcb5-e9c94e359332-istio-podinfo\") pod \"4952793b-a587-46ca-bcb5-e9c94e359332\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " Oct 30 01:50:20.992308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.992176 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-envoy\") pod \"4952793b-a587-46ca-bcb5-e9c94e359332\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " Oct 30 01:50:20.992308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.992211 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-istio-token\" (UniqueName: \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-istio-token\") pod \"4952793b-a587-46ca-bcb5-e9c94e359332\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " Oct 30 01:50:20.992308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.992259 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-credential-socket\") pod \"4952793b-a587-46ca-bcb5-e9c94e359332\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " Oct 30 01:50:20.992308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:20.992303 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-certs\") pod \"4952793b-a587-46ca-bcb5-e9c94e359332\" (UID: \"4952793b-a587-46ca-bcb5-e9c94e359332\") " Oct 30 01:50:21.030085 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.029694 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-data" pod "4952793b-a587-46ca-bcb5-e9c94e359332" (UID: "4952793b-a587-46ca-bcb5-e9c94e359332"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:50:21.036565 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.036291 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-socket" pod "4952793b-a587-46ca-bcb5-e9c94e359332" (UID: "4952793b-a587-46ca-bcb5-e9c94e359332"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:50:21.036565 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.036528 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-credential-socket" pod "4952793b-a587-46ca-bcb5-e9c94e359332" (UID: "4952793b-a587-46ca-bcb5-e9c94e359332"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:50:21.037104 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.037044 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-certs" pod "4952793b-a587-46ca-bcb5-e9c94e359332" (UID: "4952793b-a587-46ca-bcb5-e9c94e359332"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:50:21.043307 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.042401 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4952793b-a587-46ca-bcb5-e9c94e359332-istio-podinfo" pod "4952793b-a587-46ca-bcb5-e9c94e359332" (UID: "4952793b-a587-46ca-bcb5-e9c94e359332"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:50:21.043307 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.043129 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-kube-api-access-9nwlm" pod "4952793b-a587-46ca-bcb5-e9c94e359332" (UID: "4952793b-a587-46ca-bcb5-e9c94e359332"). InnerVolumeSpecName "kube-api-access-9nwlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:50:21.043307 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.043265 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-envoy" pod "4952793b-a587-46ca-bcb5-e9c94e359332" (UID: "4952793b-a587-46ca-bcb5-e9c94e359332"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:50:21.045685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.045179 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-istio-token" pod "4952793b-a587-46ca-bcb5-e9c94e359332" (UID: "4952793b-a587-46ca-bcb5-e9c94e359332"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:50:21.045685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.045624 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4952793b-a587-46ca-bcb5-e9c94e359332-istiod-ca-cert" pod "4952793b-a587-46ca-bcb5-e9c94e359332" (UID: "4952793b-a587-46ca-bcb5-e9c94e359332"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:50:21.099419 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.095409 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:21.099419 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.095442 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:21.099419 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.095464 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4952793b-a587-46ca-bcb5-e9c94e359332-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:21.099419 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.095479 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:21.099419 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.095490 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:21.099419 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.095500 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:21.099419 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.095510 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4952793b-a587-46ca-bcb5-e9c94e359332-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:21.099419 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.095522 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4952793b-a587-46ca-bcb5-e9c94e359332-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:21.099419 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.095533 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nwlm\" (UniqueName: \"kubernetes.io/projected/4952793b-a587-46ca-bcb5-e9c94e359332-kube-api-access-9nwlm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:50:21.378190 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:50:21.378157 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ff2a708_3621_4666_9789_c1e99acf4883.slice/crio-c531a19ecceb6eb4c5f4e00d6fdb32f02257d3b643adec54f397260e93bd26a2 WatchSource:0}: Error finding container c531a19ecceb6eb4c5f4e00d6fdb32f02257d3b643adec54f397260e93bd26a2: Status 404 returned error can't find the container with id c531a19ecceb6eb4c5f4e00d6fdb32f02257d3b643adec54f397260e93bd26a2 Oct 30 01:50:21.378853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.378824 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc"] Oct 30 01:50:21.512548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.512174 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:21.526067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.526033 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:21.758120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.758091 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" Oct 30 01:50:21.758683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.758119 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh" event={"ID":"4952793b-a587-46ca-bcb5-e9c94e359332","Type":"ContainerDied","Data":"5aeb3f69c5143128543c38e767ac5bbf3a8b5bf1486ef1fd5a518e44510ec046"} Oct 30 01:50:21.758683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.758185 5055 scope.go:122] "RemoveContainer" containerID="ed65beb62963ad9eeefcfb4566fdbb95a2fb19ea284b9f045fcda5edfbaaa4f7" Oct 30 01:50:21.760123 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.760091 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" event={"ID":"5ff2a708-3621-4666-9789-c1e99acf4883","Type":"ContainerStarted","Data":"45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6"} Oct 30 01:50:21.760246 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.760136 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" event={"ID":"5ff2a708-3621-4666-9789-c1e99acf4883","Type":"ContainerStarted","Data":"c531a19ecceb6eb4c5f4e00d6fdb32f02257d3b643adec54f397260e93bd26a2"} Oct 30 01:50:21.775424 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.775397 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:50:21.813401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.813307 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" podStartSLOduration=1.813290665 podStartE2EDuration="1.813290665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:21.812448255 +0000 UTC m=+1989.133405294" watchObservedRunningTime="2026-10-30 01:50:21.813290665 +0000 UTC m=+1989.134247706" Oct 30 01:50:21.854548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.854373 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh"] Oct 30 01:50:21.864279 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.864252 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-389dba-openshift-default-6875794477-rb6gh"] Oct 30 01:50:21.926795 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:21.926759 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:22.827307 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:22.824965 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" path="/var/lib/kubelet/pods/4952793b-a587-46ca-bcb5-e9c94e359332/volumes" Oct 30 01:50:30.926384 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:30.926286 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:30.940029 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:30.939997 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:31.864971 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:31.864937 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:50:36.304747 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.304713 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r"] Oct 30 01:50:36.305364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.305349 5055 cpu_manager.go:414] "containerMap: removing container" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" Oct 30 01:50:36.305364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.305365 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" Oct 30 01:50:36.305485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.305382 5055 cpu_manager.go:414] "containerMap: removing container" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" Oct 30 01:50:36.305485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.305387 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" Oct 30 01:50:36.305558 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.305518 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" Oct 30 01:50:36.309275 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.309254 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.325092 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.325068 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-264bdd-openshift-default-dockercfg-8qt8s\"" Oct 30 01:50:36.341370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.341317 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r"] Oct 30 01:50:36.439280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.439246 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.439539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.439302 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-podinfo\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.439539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.439381 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-certs\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.439539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.439416 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-token\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.439539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.439449 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-credential-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.439539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.439495 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istiod-ca-cert\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.439539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.439523 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-envoy\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.439884 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.439596 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch9xh\" (UniqueName: \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-kube-api-access-ch9xh\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.439884 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.439625 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-data\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541307 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch9xh\" (UniqueName: \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-kube-api-access-ch9xh\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541370 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-data\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541485 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541515 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-podinfo\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541532 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-certs\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541579 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541559 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-token\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541597 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-credential-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541655 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istiod-ca-cert\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541746 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-envoy\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541765 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-data\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.541889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.541860 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.542163 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.542113 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-credential-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.542255 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.542231 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-certs\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.542519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.542500 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istiod-ca-cert\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.544115 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.544099 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-envoy\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.544311 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.544294 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-podinfo\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.561213 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.561151 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-token\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.561368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.561282 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch9xh\" (UniqueName: \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-kube-api-access-ch9xh\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.625272 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.625238 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:36.950733 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:50:36.950689 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod101455c3_027f_4c48_b8d9_6691e5bb3f1f.slice/crio-59b716df81d548b60643dfba9635bbe8d65d0b79b3c22577a04ba8a3ff3f43ff WatchSource:0}: Error finding container 59b716df81d548b60643dfba9635bbe8d65d0b79b3c22577a04ba8a3ff3f43ff: Status 404 returned error can't find the container with id 59b716df81d548b60643dfba9635bbe8d65d0b79b3c22577a04ba8a3ff3f43ff Oct 30 01:50:36.952688 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.952659 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r"] Oct 30 01:50:36.954025 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.953993 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:50:36.954134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.954058 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:50:36.954134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:36.954086 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:50:37.903734 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:37.903692 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" event={"ID":"101455c3-027f-4c48-b8d9-6691e5bb3f1f","Type":"ContainerStarted","Data":"7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2"} Oct 30 01:50:37.904278 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:37.903760 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" event={"ID":"101455c3-027f-4c48-b8d9-6691e5bb3f1f","Type":"ContainerStarted","Data":"59b716df81d548b60643dfba9635bbe8d65d0b79b3c22577a04ba8a3ff3f43ff"} Oct 30 01:50:37.981354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:37.975154 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" podStartSLOduration=1.975135163 podStartE2EDuration="1.975135163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:50:37.973889185 +0000 UTC m=+2005.294846227" watchObservedRunningTime="2026-10-30 01:50:37.975135163 +0000 UTC m=+2005.296092204" Oct 30 01:50:38.626120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:38.626080 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:38.628047 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:38.628026 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 01:50:38.628120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:38.628075 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 01:50:39.625643 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:39.625609 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 01:50:39.626048 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:39.625661 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 01:50:40.626175 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:40.626135 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 01:50:40.626656 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:40.626189 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 01:50:41.625855 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:41.625813 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 01:50:41.626062 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:41.625882 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 01:50:42.626287 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:42.626247 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 01:50:42.626695 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:42.626318 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 01:50:43.626241 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:43.626198 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 30 01:50:43.626456 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:43.626281 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 30 01:50:44.631134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:44.631099 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:44.631668 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:44.631638 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:44.632422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:44.632403 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 01:50:45.579346 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:50:45.579285 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc"] Oct 30 01:50:49.220562 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:50:49.220521 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:51:00.608624 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:00.608529 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" podUID="5ff2a708-3621-4666-9789-c1e99acf4883" containerName="main" containerID="cri-o://45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6" gracePeriod=45 Oct 30 01:51:00.900134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:00.900101 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:51:01.027616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.027576 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-model-cache\") pod \"5ff2a708-3621-4666-9789-c1e99acf4883\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " Oct 30 01:51:01.027904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.027641 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-tmp-dir\") pod \"5ff2a708-3621-4666-9789-c1e99acf4883\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " Oct 30 01:51:01.027904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.027676 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-home\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-home\") pod \"5ff2a708-3621-4666-9789-c1e99acf4883\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " Oct 30 01:51:01.027904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.027728 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff2a708-3621-4666-9789-c1e99acf4883-kube-api-access-9qf6l\" (UniqueName: \"kubernetes.io/projected/5ff2a708-3621-4666-9789-c1e99acf4883-kube-api-access-9qf6l\") pod \"5ff2a708-3621-4666-9789-c1e99acf4883\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " Oct 30 01:51:01.027904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.027880 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-dshm\") pod \"5ff2a708-3621-4666-9789-c1e99acf4883\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " Oct 30 01:51:01.028150 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.027968 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-home" pod "5ff2a708-3621-4666-9789-c1e99acf4883" (UID: "5ff2a708-3621-4666-9789-c1e99acf4883"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:51:01.028150 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.027983 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-tmp-dir" pod "5ff2a708-3621-4666-9789-c1e99acf4883" (UID: "5ff2a708-3621-4666-9789-c1e99acf4883"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:51:01.028150 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.028097 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ff2a708-3621-4666-9789-c1e99acf4883-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff2a708-3621-4666-9789-c1e99acf4883-tls-certs\") pod \"5ff2a708-3621-4666-9789-c1e99acf4883\" (UID: \"5ff2a708-3621-4666-9789-c1e99acf4883\") " Oct 30 01:51:01.028507 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.028452 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-model-cache" pod "5ff2a708-3621-4666-9789-c1e99acf4883" (UID: "5ff2a708-3621-4666-9789-c1e99acf4883"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:51:01.029055 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.029034 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:51:01.029140 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.029060 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:51:01.029140 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.029075 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:51:01.030461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.030440 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-dshm" pod "5ff2a708-3621-4666-9789-c1e99acf4883" (UID: "5ff2a708-3621-4666-9789-c1e99acf4883"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:51:01.030664 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.030642 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff2a708-3621-4666-9789-c1e99acf4883-kube-api-access-9qf6l" pod "5ff2a708-3621-4666-9789-c1e99acf4883" (UID: "5ff2a708-3621-4666-9789-c1e99acf4883"). InnerVolumeSpecName "kube-api-access-9qf6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:51:01.030748 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.030726 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff2a708-3621-4666-9789-c1e99acf4883-tls-certs" pod "5ff2a708-3621-4666-9789-c1e99acf4883" (UID: "5ff2a708-3621-4666-9789-c1e99acf4883"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:51:01.130810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.130727 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ff2a708-3621-4666-9789-c1e99acf4883-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:51:01.130810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.130759 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qf6l\" (UniqueName: \"kubernetes.io/projected/5ff2a708-3621-4666-9789-c1e99acf4883-kube-api-access-9qf6l\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:51:01.130810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.130768 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ff2a708-3621-4666-9789-c1e99acf4883-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:51:01.191860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.191590 5055 generic.go:356] "Generic (PLEG): container finished" podID="5ff2a708-3621-4666-9789-c1e99acf4883" containerID="45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6" exitCode=0 Oct 30 01:51:01.191860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.191640 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" Oct 30 01:51:01.191860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.191637 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" event={"ID":"5ff2a708-3621-4666-9789-c1e99acf4883","Type":"ContainerDied","Data":"45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6"} Oct 30 01:51:01.191860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.191693 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" event={"ID":"5ff2a708-3621-4666-9789-c1e99acf4883","Type":"ContainerDied","Data":"c531a19ecceb6eb4c5f4e00d6fdb32f02257d3b643adec54f397260e93bd26a2"} Oct 30 01:51:01.191860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.191716 5055 scope.go:122] "RemoveContainer" containerID="45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6" Oct 30 01:51:01.207102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.207079 5055 scope.go:122] "RemoveContainer" containerID="45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6" Oct 30 01:51:01.207548 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:51:01.207524 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6\": container with ID starting with 45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6 not found: ID does not exist" containerID="45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6" Oct 30 01:51:01.207651 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.207562 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6"} err="failed to get container status \"45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6\": rpc error: code = NotFound desc = could not find container \"45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6\": container with ID starting with 45043f6950362e661e27bd5787e96f1b01ceb3bd40736aa375a073263343c2b6 not found: ID does not exist" Oct 30 01:51:01.244043 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.244011 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc"] Oct 30 01:51:01.247219 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.247191 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc"] Oct 30 01:51:01.855413 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:01.855312 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-noten-36c949/noten-model-36c949-kserve-5b977d6597-4c5wc" podUID="5ff2a708-3621-4666-9789-c1e99acf4883" containerName="main" probeResult="failure" output="Get \"https://10.132.0.150:8000/ready\": context deadline exceeded" Oct 30 01:51:02.828193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:02.828157 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff2a708-3621-4666-9789-c1e99acf4883" path="/var/lib/kubelet/pods/5ff2a708-3621-4666-9789-c1e99acf4883/volumes" Oct 30 01:51:19.221661 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:51:19.221615 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:51:28.609371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.609312 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7495-272bfb"] Oct 30 01:51:28.610302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.610278 5055 cpu_manager.go:414] "containerMap: removing container" podUID="5ff2a708-3621-4666-9789-c1e99acf4883" containerName="main" Oct 30 01:51:28.610380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.610307 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff2a708-3621-4666-9789-c1e99acf4883" containerName="main" Oct 30 01:51:28.610492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.610481 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff2a708-3621-4666-9789-c1e99acf4883" containerName="main" Oct 30 01:51:28.610529 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.610498 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="4952793b-a587-46ca-bcb5-e9c94e359332" containerName="istio-proxy" Oct 30 01:51:28.615501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.615483 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-272bfb" Oct 30 01:51:28.630715 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.630691 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7495-272bfb"] Oct 30 01:51:28.639256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.639226 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pm9kh\"" Oct 30 01:51:28.671149 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.671116 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrddk\" (UniqueName: \"kubernetes.io/projected/8409da1d-5b4e-4a2a-881a-7e756cf63ec9-kube-api-access-wrddk\") pod \"test-curl-7495-272bfb\" (UID: \"8409da1d-5b4e-4a2a-881a-7e756cf63ec9\") " pod="opendatahub/test-curl-7495-272bfb" Oct 30 01:51:28.772201 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.772169 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrddk\" (UniqueName: \"kubernetes.io/projected/8409da1d-5b4e-4a2a-881a-7e756cf63ec9-kube-api-access-wrddk\") pod \"test-curl-7495-272bfb\" (UID: \"8409da1d-5b4e-4a2a-881a-7e756cf63ec9\") " pod="opendatahub/test-curl-7495-272bfb" Oct 30 01:51:28.792280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.792248 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrddk\" (UniqueName: \"kubernetes.io/projected/8409da1d-5b4e-4a2a-881a-7e756cf63ec9-kube-api-access-wrddk\") pod \"test-curl-7495-272bfb\" (UID: \"8409da1d-5b4e-4a2a-881a-7e756cf63ec9\") " pod="opendatahub/test-curl-7495-272bfb" Oct 30 01:51:28.928243 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:28.928162 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-272bfb" Oct 30 01:51:29.242951 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:51:29.242902 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8409da1d_5b4e_4a2a_881a_7e756cf63ec9.slice/crio-bcf343501a80ba8544fddddbb23d947ff3c8ddaeb7f481fc09941861c4a2a6fa WatchSource:0}: Error finding container bcf343501a80ba8544fddddbb23d947ff3c8ddaeb7f481fc09941861c4a2a6fa: Status 404 returned error can't find the container with id bcf343501a80ba8544fddddbb23d947ff3c8ddaeb7f481fc09941861c4a2a6fa Oct 30 01:51:29.244378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:29.244349 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7495-272bfb"] Oct 30 01:51:29.415899 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:29.415866 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7495-272bfb" event={"ID":"8409da1d-5b4e-4a2a-881a-7e756cf63ec9","Type":"ContainerStarted","Data":"6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a"} Oct 30 01:51:29.416079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:29.415915 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7495-272bfb" event={"ID":"8409da1d-5b4e-4a2a-881a-7e756cf63ec9","Type":"ContainerStarted","Data":"bcf343501a80ba8544fddddbb23d947ff3c8ddaeb7f481fc09941861c4a2a6fa"} Oct 30 01:51:29.454119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:29.454067 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7495-272bfb" podStartSLOduration=1.454052791 podStartE2EDuration="1.454052791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:29.451755419 +0000 UTC m=+2056.772712459" watchObservedRunningTime="2026-10-30 01:51:29.454052791 +0000 UTC m=+2056.775009830" Oct 30 01:51:30.396454 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:30.396381 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7495-272bfb"] Oct 30 01:51:30.400984 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:30.400954 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7495-272bfb"] Oct 30 01:51:31.438087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:31.438043 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7495-272bfb" podUID="8409da1d-5b4e-4a2a-881a-7e756cf63ec9" containerName="test-curl-7495-272bfb" containerID="cri-o://6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a" gracePeriod=2 Oct 30 01:51:33.691344 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:33.691302 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-272bfb" Oct 30 01:51:33.771203 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:33.771165 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8409da1d-5b4e-4a2a-881a-7e756cf63ec9-kube-api-access-wrddk\" (UniqueName: \"kubernetes.io/projected/8409da1d-5b4e-4a2a-881a-7e756cf63ec9-kube-api-access-wrddk\") pod \"8409da1d-5b4e-4a2a-881a-7e756cf63ec9\" (UID: \"8409da1d-5b4e-4a2a-881a-7e756cf63ec9\") " Oct 30 01:51:33.774125 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:33.774091 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8409da1d-5b4e-4a2a-881a-7e756cf63ec9-kube-api-access-wrddk" pod "8409da1d-5b4e-4a2a-881a-7e756cf63ec9" (UID: "8409da1d-5b4e-4a2a-881a-7e756cf63ec9"). InnerVolumeSpecName "kube-api-access-wrddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:51:33.873241 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:33.873191 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrddk\" (UniqueName: \"kubernetes.io/projected/8409da1d-5b4e-4a2a-881a-7e756cf63ec9-kube-api-access-wrddk\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:51:34.464587 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:34.464556 5055 generic.go:356] "Generic (PLEG): container finished" podID="8409da1d-5b4e-4a2a-881a-7e756cf63ec9" containerID="6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a" exitCode=137 Oct 30 01:51:34.464810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:34.464621 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7495-272bfb" Oct 30 01:51:34.464810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:34.464659 5055 scope.go:122] "RemoveContainer" containerID="6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a" Oct 30 01:51:34.476791 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:34.476765 5055 scope.go:122] "RemoveContainer" containerID="6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a" Oct 30 01:51:34.477593 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:51:34.477506 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a\": container with ID starting with 6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a not found: ID does not exist" containerID="6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a" Oct 30 01:51:34.477593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:34.477550 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a"} err="failed to get container status \"6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a\": rpc error: code = NotFound desc = could not find container \"6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a\": container with ID starting with 6fb4c778237e9d150c21a09648c9734d04a0c8e72f33f69df76a899f9f8a7d9a not found: ID does not exist" Oct 30 01:51:34.491633 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:51:34.491600 5055 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7495-272bfb\" is forbidden: User \"system:node:ip-10-0-134-254.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-254.ec2.internal' and this object" podUID="8409da1d-5b4e-4a2a-881a-7e756cf63ec9" pod="opendatahub/test-curl-7495-272bfb" Oct 30 01:51:34.823011 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:34.822933 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8409da1d-5b4e-4a2a-881a-7e756cf63ec9" path="/var/lib/kubelet/pods/8409da1d-5b4e-4a2a-881a-7e756cf63ec9/volumes" Oct 30 01:51:44.987649 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:44.987614 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm"] Oct 30 01:51:49.223797 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:51:49.223756 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:51:57.733677 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.733642 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk"] Oct 30 01:51:57.734296 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.734283 5055 cpu_manager.go:414] "containerMap: removing container" podUID="8409da1d-5b4e-4a2a-881a-7e756cf63ec9" containerName="test-curl-7495-272bfb" Oct 30 01:51:57.734378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.734297 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8409da1d-5b4e-4a2a-881a-7e756cf63ec9" containerName="test-curl-7495-272bfb" Oct 30 01:51:57.734443 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.734431 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="8409da1d-5b4e-4a2a-881a-7e756cf63ec9" containerName="test-curl-7495-272bfb" Oct 30 01:51:57.739590 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.739571 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.788365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.786071 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-5fb950-serving-cert\"" Oct 30 01:51:57.795656 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.795625 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.795976 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.795954 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.796337 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.796303 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nssc\" (UniqueName: \"kubernetes.io/projected/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-kube-api-access-8nssc\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.796523 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.796497 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.802724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.798135 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk"] Oct 30 01:51:57.897643 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.897602 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.897902 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.897736 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.897902 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.897885 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nssc\" (UniqueName: \"kubernetes.io/projected/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-kube-api-access-8nssc\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.898019 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.897968 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.899175 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.899132 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.901550 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.901488 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.902391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.902362 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:57.919290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:57.919254 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nssc\" (UniqueName: \"kubernetes.io/projected/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-kube-api-access-8nssc\") pod \"maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:58.054611 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.054491 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:58.311435 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.311341 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr"] Oct 30 01:51:58.311627 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.311559 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" containerID="cri-o://b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981" gracePeriod=30 Oct 30 01:51:58.314512 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.314480 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd"] Oct 30 01:51:58.314760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.314727 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" containerID="cri-o://d4c68c868f28f4b9ea4e54a20bb39b80244e3faf8ed35345b61f37b4f1f58090" gracePeriod=30 Oct 30 01:51:58.317941 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.317912 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl"] Oct 30 01:51:58.323830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.323797 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn"] Oct 30 01:51:58.324073 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.324046 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerName="istio-proxy" containerID="cri-o://b3f3fc8af7c7ebfb7b61426f2dc53a146dfc1454907dd67bacf27490a146ee83" gracePeriod=30 Oct 30 01:51:58.398967 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.398936 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk"] Oct 30 01:51:58.399577 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:51:58.399545 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1c5a9d_8deb_4ebc_bc1e_ad12960134a2.slice/crio-70dd6e8fc5736275d36aaa55e5071a8eabcfdd52f89aa39f04c0e8b4d6cb18f3 WatchSource:0}: Error finding container 70dd6e8fc5736275d36aaa55e5071a8eabcfdd52f89aa39f04c0e8b4d6cb18f3: Status 404 returned error can't find the container with id 70dd6e8fc5736275d36aaa55e5071a8eabcfdd52f89aa39f04c0e8b4d6cb18f3 Oct 30 01:51:58.655422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.655285 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" event={"ID":"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2","Type":"ContainerStarted","Data":"70dd6e8fc5736275d36aaa55e5071a8eabcfdd52f89aa39f04c0e8b4d6cb18f3"} Oct 30 01:51:58.655607 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:58.655442 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" containerID="cri-o://6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6" gracePeriod=30 Oct 30 01:51:59.664819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:59.664783 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" event={"ID":"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2","Type":"ContainerStarted","Data":"8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4"} Oct 30 01:51:59.665364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:59.664937 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:51:59.708119 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:51:59.708055 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" podStartSLOduration=2.311023355 podStartE2EDuration="2.708034651s" totalImagesPullingTime="397.011296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:51:59.706935647 +0000 UTC m=+2087.027892679" watchObservedRunningTime="2026-10-30 01:51:59.708034651 +0000 UTC m=+2087.028991692" Oct 30 01:52:00.030760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.030721 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" podUID="2dbbeb20-ed61-433a-a0f1-11bcf3948b69" containerName="main" containerID="cri-o://df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca" gracePeriod=45 Oct 30 01:52:00.352347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.352307 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:52:00.453829 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.453786 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tls-certs\") pod \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " Oct 30 01:52:00.454036 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.453864 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tmp-dir\") pod \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " Oct 30 01:52:00.454036 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.453969 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-model-cache\") pod \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " Oct 30 01:52:00.454156 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.454089 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-home\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-home\") pod \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " Oct 30 01:52:00.454156 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.454119 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-kube-api-access-lx72m\" (UniqueName: \"kubernetes.io/projected/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-kube-api-access-lx72m\") pod \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " Oct 30 01:52:00.454260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.454174 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tmp-dir" pod "2dbbeb20-ed61-433a-a0f1-11bcf3948b69" (UID: "2dbbeb20-ed61-433a-a0f1-11bcf3948b69"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:00.454260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.454230 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-dshm\") pod \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\" (UID: \"2dbbeb20-ed61-433a-a0f1-11bcf3948b69\") " Oct 30 01:52:00.454424 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.454400 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-home" pod "2dbbeb20-ed61-433a-a0f1-11bcf3948b69" (UID: "2dbbeb20-ed61-433a-a0f1-11bcf3948b69"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:00.454730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.454709 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-model-cache" pod "2dbbeb20-ed61-433a-a0f1-11bcf3948b69" (UID: "2dbbeb20-ed61-433a-a0f1-11bcf3948b69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:00.454854 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.454837 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:00.454903 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.454860 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:00.454903 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.454875 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:00.456920 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.456894 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-dshm" pod "2dbbeb20-ed61-433a-a0f1-11bcf3948b69" (UID: "2dbbeb20-ed61-433a-a0f1-11bcf3948b69"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:00.457672 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.457652 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tls-certs" pod "2dbbeb20-ed61-433a-a0f1-11bcf3948b69" (UID: "2dbbeb20-ed61-433a-a0f1-11bcf3948b69"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:52:00.457772 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.457690 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-kube-api-access-lx72m" pod "2dbbeb20-ed61-433a-a0f1-11bcf3948b69" (UID: "2dbbeb20-ed61-433a-a0f1-11bcf3948b69"). InnerVolumeSpecName "kube-api-access-lx72m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:00.556085 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.556044 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:00.556085 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.556081 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx72m\" (UniqueName: \"kubernetes.io/projected/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-kube-api-access-lx72m\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:00.556085 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.556091 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dbbeb20-ed61-433a-a0f1-11bcf3948b69-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:00.674111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.674018 5055 generic.go:356] "Generic (PLEG): container finished" podID="2dbbeb20-ed61-433a-a0f1-11bcf3948b69" containerID="df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca" exitCode=0 Oct 30 01:52:00.674111 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.674082 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" Oct 30 01:52:00.674699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.674109 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" event={"ID":"2dbbeb20-ed61-433a-a0f1-11bcf3948b69","Type":"ContainerDied","Data":"df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca"} Oct 30 01:52:00.674699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.674162 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" event={"ID":"2dbbeb20-ed61-433a-a0f1-11bcf3948b69","Type":"ContainerDied","Data":"8976fba551da8a1b3c0867b58628ae42d773435946b09f565f8ec876354613d9"} Oct 30 01:52:00.674699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.674188 5055 scope.go:122] "RemoveContainer" containerID="df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca" Oct 30 01:52:00.687739 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.687716 5055 scope.go:122] "RemoveContainer" containerID="df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca" Oct 30 01:52:00.688092 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:00.688068 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca\": container with ID starting with df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca not found: ID does not exist" containerID="df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca" Oct 30 01:52:00.688215 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.688097 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca"} err="failed to get container status \"df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca\": rpc error: code = NotFound desc = could not find container \"df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca\": container with ID starting with df7cee15cbb9453b74ea8abf4715cdab5ee61f32c21f3f189a5a8a889f2422ca not found: ID does not exist" Oct 30 01:52:00.729694 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.729653 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm"] Oct 30 01:52:00.740116 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.740086 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm"] Oct 30 01:52:00.822260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:00.822225 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dbbeb20-ed61-433a-a0f1-11bcf3948b69" path="/var/lib/kubelet/pods/2dbbeb20-ed61-433a-a0f1-11bcf3948b69/volumes" Oct 30 01:52:01.276950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:01.276908 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-396f8e/auth-test-model-396f8e-kserve-7db4f564c7-ms8bm" podUID="2dbbeb20-ed61-433a-a0f1-11bcf3948b69" containerName="main" probeResult="failure" output="Get \"https://10.132.0.143:8000/ready\": context deadline exceeded" Oct 30 01:52:03.647099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.647070 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:52:03.706861 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.706826 5055 generic.go:356] "Generic (PLEG): container finished" podID="dcfcf547-3b5f-4907-b88e-392969c79158" containerID="d4c68c868f28f4b9ea4e54a20bb39b80244e3faf8ed35345b61f37b4f1f58090" exitCode=0 Oct 30 01:52:03.707067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.706965 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" event={"ID":"dcfcf547-3b5f-4907-b88e-392969c79158","Type":"ContainerDied","Data":"d4c68c868f28f4b9ea4e54a20bb39b80244e3faf8ed35345b61f37b4f1f58090"} Oct 30 01:52:03.707067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.707007 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" event={"ID":"dcfcf547-3b5f-4907-b88e-392969c79158","Type":"ContainerDied","Data":"372508e8b88f3aa75720baf35f03a995e4f28e028f1c04d3f72de12b01308a9a"} Oct 30 01:52:03.707067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.707024 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="372508e8b88f3aa75720baf35f03a995e4f28e028f1c04d3f72de12b01308a9a" Oct 30 01:52:03.708231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.708112 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beafde80-c491-4977-8a5e-8abe3827fb07-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beafde80-c491-4977-8a5e-8abe3827fb07-istio-podinfo\") pod \"beafde80-c491-4977-8a5e-8abe3827fb07\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " Oct 30 01:52:03.708403 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.708360 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-certs\") pod \"beafde80-c491-4977-8a5e-8abe3827fb07\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " Oct 30 01:52:03.708469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.708411 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beafde80-c491-4977-8a5e-8abe3827fb07-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beafde80-c491-4977-8a5e-8abe3827fb07-istiod-ca-cert\") pod \"beafde80-c491-4977-8a5e-8abe3827fb07\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " Oct 30 01:52:03.708469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.708457 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-istio-token\" (UniqueName: \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-istio-token\") pod \"beafde80-c491-4977-8a5e-8abe3827fb07\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " Oct 30 01:52:03.708572 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.708506 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-data\") pod \"beafde80-c491-4977-8a5e-8abe3827fb07\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " Oct 30 01:52:03.708642 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.708631 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-socket\") pod \"beafde80-c491-4977-8a5e-8abe3827fb07\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " Oct 30 01:52:03.708696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.708676 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-kube-api-access-4hdx6\" (UniqueName: \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-kube-api-access-4hdx6\") pod \"beafde80-c491-4977-8a5e-8abe3827fb07\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " Oct 30 01:52:03.708752 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.708700 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-credential-socket\") pod \"beafde80-c491-4977-8a5e-8abe3827fb07\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " Oct 30 01:52:03.709052 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.709029 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-credential-socket" pod "beafde80-c491-4977-8a5e-8abe3827fb07" (UID: "beafde80-c491-4977-8a5e-8abe3827fb07"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.709228 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.709211 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-data" pod "beafde80-c491-4977-8a5e-8abe3827fb07" (UID: "beafde80-c491-4977-8a5e-8abe3827fb07"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.711368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.709386 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-socket" pod "beafde80-c491-4977-8a5e-8abe3827fb07" (UID: "beafde80-c491-4977-8a5e-8abe3827fb07"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.711368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.709849 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-certs" pod "beafde80-c491-4977-8a5e-8abe3827fb07" (UID: "beafde80-c491-4977-8a5e-8abe3827fb07"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.711368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.710005 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-envoy\") pod \"beafde80-c491-4977-8a5e-8abe3827fb07\" (UID: \"beafde80-c491-4977-8a5e-8abe3827fb07\") " Oct 30 01:52:03.711368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.710308 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beafde80-c491-4977-8a5e-8abe3827fb07-istiod-ca-cert" pod "beafde80-c491-4977-8a5e-8abe3827fb07" (UID: "beafde80-c491-4977-8a5e-8abe3827fb07"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:03.711368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.711092 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.711368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.711114 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beafde80-c491-4977-8a5e-8abe3827fb07-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.711368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.711127 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.711368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.711140 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.711368 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.711155 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.712842 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.712805 5055 generic.go:356] "Generic (PLEG): container finished" podID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerID="b3f3fc8af7c7ebfb7b61426f2dc53a146dfc1454907dd67bacf27490a146ee83" exitCode=0 Oct 30 01:52:03.712944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.712847 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" event={"ID":"ea0b9039-8705-4ba7-8942-991b5fd80d9b","Type":"ContainerDied","Data":"b3f3fc8af7c7ebfb7b61426f2dc53a146dfc1454907dd67bacf27490a146ee83"} Oct 30 01:52:03.714776 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.714748 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beafde80-c491-4977-8a5e-8abe3827fb07-istio-podinfo" pod "beafde80-c491-4977-8a5e-8abe3827fb07" (UID: "beafde80-c491-4977-8a5e-8abe3827fb07"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:03.715143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.715118 5055 generic.go:356] "Generic (PLEG): container finished" podID="beafde80-c491-4977-8a5e-8abe3827fb07" containerID="b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981" exitCode=0 Oct 30 01:52:03.715207 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.715183 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" event={"ID":"beafde80-c491-4977-8a5e-8abe3827fb07","Type":"ContainerDied","Data":"b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981"} Oct 30 01:52:03.715252 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.715232 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" event={"ID":"beafde80-c491-4977-8a5e-8abe3827fb07","Type":"ContainerDied","Data":"9b61e471eedf38b5eb03f696abde707c0d9a8443efd54f7c5e250c2b21807e7c"} Oct 30 01:52:03.715286 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.715263 5055 scope.go:122] "RemoveContainer" containerID="b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981" Oct 30 01:52:03.715536 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.715518 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr" Oct 30 01:52:03.715706 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.715683 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-kube-api-access-4hdx6" pod "beafde80-c491-4977-8a5e-8abe3827fb07" (UID: "beafde80-c491-4977-8a5e-8abe3827fb07"). InnerVolumeSpecName "kube-api-access-4hdx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:03.715788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.715772 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-envoy" pod "beafde80-c491-4977-8a5e-8abe3827fb07" (UID: "beafde80-c491-4977-8a5e-8abe3827fb07"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.717496 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.717472 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-istio-token" pod "beafde80-c491-4977-8a5e-8abe3827fb07" (UID: "beafde80-c491-4977-8a5e-8abe3827fb07"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:03.770483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.770134 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:52:03.775445 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.775415 5055 scope.go:122] "RemoveContainer" containerID="b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981" Oct 30 01:52:03.775848 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:03.775777 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981\": container with ID starting with b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981 not found: ID does not exist" containerID="b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981" Oct 30 01:52:03.775848 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.775820 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981"} err="failed to get container status \"b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981\": rpc error: code = NotFound desc = could not find container \"b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981\": container with ID starting with b206770dcfecfcf47cbe67f62c43f0337ef67168998fb7c4a8c0670686ee7981 not found: ID does not exist" Oct 30 01:52:03.777156 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.777132 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:52:03.812502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.812466 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hdx6\" (UniqueName: \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-kube-api-access-4hdx6\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.812502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.812503 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beafde80-c491-4977-8a5e-8abe3827fb07-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.812779 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.812521 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beafde80-c491-4977-8a5e-8abe3827fb07-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.812779 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.812539 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beafde80-c491-4977-8a5e-8abe3827fb07-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.913933 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.913826 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istiod-ca-cert\") pod \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " Oct 30 01:52:03.913933 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.913888 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-certs\") pod \"dcfcf547-3b5f-4907-b88e-392969c79158\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " Oct 30 01:52:03.914172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.913942 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-envoy\") pod \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " Oct 30 01:52:03.914172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914029 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-credential-socket\") pod \"dcfcf547-3b5f-4907-b88e-392969c79158\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " Oct 30 01:52:03.914172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914096 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-kube-api-access-tc4bs\" (UniqueName: \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-kube-api-access-tc4bs\") pod \"dcfcf547-3b5f-4907-b88e-392969c79158\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " Oct 30 01:52:03.914172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914126 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-certs\") pod \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " Oct 30 01:52:03.914172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914151 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcfcf547-3b5f-4907-b88e-392969c79158-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfcf547-3b5f-4907-b88e-392969c79158-istiod-ca-cert\") pod \"dcfcf547-3b5f-4907-b88e-392969c79158\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " Oct 30 01:52:03.914501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914191 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-kube-api-access-zlncs\" (UniqueName: \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-kube-api-access-zlncs\") pod \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " Oct 30 01:52:03.914501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914236 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-envoy\") pod \"dcfcf547-3b5f-4907-b88e-392969c79158\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " Oct 30 01:52:03.914501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914262 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-socket\") pod \"dcfcf547-3b5f-4907-b88e-392969c79158\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " Oct 30 01:52:03.914501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914298 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-token\" (UniqueName: \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-token\") pod \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " Oct 30 01:52:03.914501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914409 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-credential-socket\") pod \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " Oct 30 01:52:03.914501 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914459 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcfcf547-3b5f-4907-b88e-392969c79158-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfcf547-3b5f-4907-b88e-392969c79158-istio-podinfo\") pod \"dcfcf547-3b5f-4907-b88e-392969c79158\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " Oct 30 01:52:03.914898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914539 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-data\") pod \"dcfcf547-3b5f-4907-b88e-392969c79158\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " Oct 30 01:52:03.914898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914583 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-data\") pod \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " Oct 30 01:52:03.914898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914620 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-podinfo\") pod \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " Oct 30 01:52:03.914898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914762 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-istio-token\" (UniqueName: \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-istio-token\") pod \"dcfcf547-3b5f-4907-b88e-392969c79158\" (UID: \"dcfcf547-3b5f-4907-b88e-392969c79158\") " Oct 30 01:52:03.914898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914799 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-socket\") pod \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\" (UID: \"ea0b9039-8705-4ba7-8942-991b5fd80d9b\") " Oct 30 01:52:03.915159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.914930 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfcf547-3b5f-4907-b88e-392969c79158-istiod-ca-cert" pod "dcfcf547-3b5f-4907-b88e-392969c79158" (UID: "dcfcf547-3b5f-4907-b88e-392969c79158"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:03.915683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.915289 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istiod-ca-cert" pod "ea0b9039-8705-4ba7-8942-991b5fd80d9b" (UID: "ea0b9039-8705-4ba7-8942-991b5fd80d9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:03.915683 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.915596 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-data" pod "dcfcf547-3b5f-4907-b88e-392969c79158" (UID: "dcfcf547-3b5f-4907-b88e-392969c79158"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.915859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.915796 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-data" pod "ea0b9039-8705-4ba7-8942-991b5fd80d9b" (UID: "ea0b9039-8705-4ba7-8942-991b5fd80d9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.916000 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.915977 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-socket" pod "ea0b9039-8705-4ba7-8942-991b5fd80d9b" (UID: "ea0b9039-8705-4ba7-8942-991b5fd80d9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.917066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.916176 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-credential-socket" pod "dcfcf547-3b5f-4907-b88e-392969c79158" (UID: "dcfcf547-3b5f-4907-b88e-392969c79158"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.917066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.916306 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-certs" pod "dcfcf547-3b5f-4907-b88e-392969c79158" (UID: "dcfcf547-3b5f-4907-b88e-392969c79158"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.917066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.917021 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-certs" pod "ea0b9039-8705-4ba7-8942-991b5fd80d9b" (UID: "ea0b9039-8705-4ba7-8942-991b5fd80d9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.917566 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.917524 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-credential-socket" pod "ea0b9039-8705-4ba7-8942-991b5fd80d9b" (UID: "ea0b9039-8705-4ba7-8942-991b5fd80d9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.917874 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.917846 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-socket" pod "dcfcf547-3b5f-4907-b88e-392969c79158" (UID: "dcfcf547-3b5f-4907-b88e-392969c79158"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.919365 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919318 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-envoy" pod "dcfcf547-3b5f-4907-b88e-392969c79158" (UID: "dcfcf547-3b5f-4907-b88e-392969c79158"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.919675 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919651 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.919740 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919683 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcfcf547-3b5f-4907-b88e-392969c79158-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.919740 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919697 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.919740 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919711 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.919740 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919725 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.919918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919740 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.919918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919756 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.919918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919768 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.919918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919782 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.919918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919795 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.919918 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.919807 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcfcf547-3b5f-4907-b88e-392969c79158-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:03.924870 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.924824 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-kube-api-access-zlncs" pod "ea0b9039-8705-4ba7-8942-991b5fd80d9b" (UID: "ea0b9039-8705-4ba7-8942-991b5fd80d9b"). InnerVolumeSpecName "kube-api-access-zlncs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:03.925310 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.925118 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-envoy" pod "ea0b9039-8705-4ba7-8942-991b5fd80d9b" (UID: "ea0b9039-8705-4ba7-8942-991b5fd80d9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:03.925310 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.925206 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-istio-token" pod "dcfcf547-3b5f-4907-b88e-392969c79158" (UID: "dcfcf547-3b5f-4907-b88e-392969c79158"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:03.927067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.927041 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcfcf547-3b5f-4907-b88e-392969c79158-istio-podinfo" pod "dcfcf547-3b5f-4907-b88e-392969c79158" (UID: "dcfcf547-3b5f-4907-b88e-392969c79158"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:03.927552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.927526 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-token" pod "ea0b9039-8705-4ba7-8942-991b5fd80d9b" (UID: "ea0b9039-8705-4ba7-8942-991b5fd80d9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:03.929411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.929384 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-podinfo" pod "ea0b9039-8705-4ba7-8942-991b5fd80d9b" (UID: "ea0b9039-8705-4ba7-8942-991b5fd80d9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:03.930159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.930118 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-kube-api-access-tc4bs" pod "dcfcf547-3b5f-4907-b88e-392969c79158" (UID: "dcfcf547-3b5f-4907-b88e-392969c79158"). InnerVolumeSpecName "kube-api-access-tc4bs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:03.979733 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:03.979698 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:52:04.021073 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.021034 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc4bs\" (UniqueName: \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-kube-api-access-tc4bs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.021260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.021081 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlncs\" (UniqueName: \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-kube-api-access-zlncs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.021260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.021100 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.021260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.021114 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcfcf547-3b5f-4907-b88e-392969c79158-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.021260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.021128 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.021260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.021143 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcfcf547-3b5f-4907-b88e-392969c79158-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.021260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.021156 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea0b9039-8705-4ba7-8942-991b5fd80d9b-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.075246 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.075186 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr"] Oct 30 01:52:04.078378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.078341 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-xt6kr"] Oct 30 01:52:04.122140 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122097 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-credential-socket\") pod \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " Oct 30 01:52:04.122399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122165 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-envoy\") pod \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " Oct 30 01:52:04.122399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122194 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-token\" (UniqueName: \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-token\") pod \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " Oct 30 01:52:04.122399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122240 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-kube-api-access-lqmxl\" (UniqueName: \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-kube-api-access-lqmxl\") pod \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " Oct 30 01:52:04.122399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122277 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istiod-ca-cert\") pod \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " Oct 30 01:52:04.122399 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122304 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-socket\") pod \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " Oct 30 01:52:04.122685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122410 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-podinfo\") pod \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " Oct 30 01:52:04.122685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122440 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-data\") pod \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " Oct 30 01:52:04.122685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122553 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-certs\") pod \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\" (UID: \"ee3e48ad-a4ca-418f-843e-46d87af6ee94\") " Oct 30 01:52:04.122685 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122556 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-credential-socket" pod "ee3e48ad-a4ca-418f-843e-46d87af6ee94" (UID: "ee3e48ad-a4ca-418f-843e-46d87af6ee94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:04.122888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122729 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-socket" pod "ee3e48ad-a4ca-418f-843e-46d87af6ee94" (UID: "ee3e48ad-a4ca-418f-843e-46d87af6ee94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:04.122888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122768 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-data" pod "ee3e48ad-a4ca-418f-843e-46d87af6ee94" (UID: "ee3e48ad-a4ca-418f-843e-46d87af6ee94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:04.122888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.122840 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istiod-ca-cert" pod "ee3e48ad-a4ca-418f-843e-46d87af6ee94" (UID: "ee3e48ad-a4ca-418f-843e-46d87af6ee94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:04.123167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.123056 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-certs" pod "ee3e48ad-a4ca-418f-843e-46d87af6ee94" (UID: "ee3e48ad-a4ca-418f-843e-46d87af6ee94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:04.123429 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.123406 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.123519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.123435 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.123519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.123452 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.123519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.123466 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.123519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.123479 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.124768 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.124745 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-envoy" pod "ee3e48ad-a4ca-418f-843e-46d87af6ee94" (UID: "ee3e48ad-a4ca-418f-843e-46d87af6ee94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:04.124863 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.124794 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-token" pod "ee3e48ad-a4ca-418f-843e-46d87af6ee94" (UID: "ee3e48ad-a4ca-418f-843e-46d87af6ee94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:04.124985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.124957 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-kube-api-access-lqmxl" pod "ee3e48ad-a4ca-418f-843e-46d87af6ee94" (UID: "ee3e48ad-a4ca-418f-843e-46d87af6ee94"). InnerVolumeSpecName "kube-api-access-lqmxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:04.125071 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.125021 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-podinfo" pod "ee3e48ad-a4ca-418f-843e-46d87af6ee94" (UID: "ee3e48ad-a4ca-418f-843e-46d87af6ee94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:04.224835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.224804 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmxl\" (UniqueName: \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-kube-api-access-lqmxl\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.224835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.224834 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.225072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.224846 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.225072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.224855 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee3e48ad-a4ca-418f-843e-46d87af6ee94-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:04.431975 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.431942 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7486-fa2888"] Oct 30 01:52:04.432561 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432548 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerName="istio-proxy" Oct 30 01:52:04.432621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432563 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerName="istio-proxy" Oct 30 01:52:04.432621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432583 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2dbbeb20-ed61-433a-a0f1-11bcf3948b69" containerName="main" Oct 30 01:52:04.432621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432588 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dbbeb20-ed61-433a-a0f1-11bcf3948b69" containerName="main" Oct 30 01:52:04.432621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432610 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" Oct 30 01:52:04.432621 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432615 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" Oct 30 01:52:04.432778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432628 5055 cpu_manager.go:414] "containerMap: removing container" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" Oct 30 01:52:04.432778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432633 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" Oct 30 01:52:04.432778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432645 5055 cpu_manager.go:414] "containerMap: removing container" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" Oct 30 01:52:04.432778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432649 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" Oct 30 01:52:04.432778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432730 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" containerName="istio-proxy" Oct 30 01:52:04.432778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432742 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" containerName="istio-proxy" Oct 30 01:52:04.432778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432750 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dbbeb20-ed61-433a-a0f1-11bcf3948b69" containerName="main" Oct 30 01:52:04.432778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432759 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerName="istio-proxy" Oct 30 01:52:04.432778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.432765 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" containerName="istio-proxy" Oct 30 01:52:04.436267 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.436242 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-fa2888" Oct 30 01:52:04.451476 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.451448 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pm9kh\"" Oct 30 01:52:04.454250 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.454216 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-fa2888"] Oct 30 01:52:04.529125 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.529028 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnh7l\" (UniqueName: \"kubernetes.io/projected/f22a6833-bf52-4c84-9b89-16bcf3231d87-kube-api-access-mnh7l\") pod \"test-curl-7486-fa2888\" (UID: \"f22a6833-bf52-4c84-9b89-16bcf3231d87\") " pod="opendatahub/test-curl-7486-fa2888" Oct 30 01:52:04.630893 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.630851 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnh7l\" (UniqueName: \"kubernetes.io/projected/f22a6833-bf52-4c84-9b89-16bcf3231d87-kube-api-access-mnh7l\") pod \"test-curl-7486-fa2888\" (UID: \"f22a6833-bf52-4c84-9b89-16bcf3231d87\") " pod="opendatahub/test-curl-7486-fa2888" Oct 30 01:52:04.650544 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.650502 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnh7l\" (UniqueName: \"kubernetes.io/projected/f22a6833-bf52-4c84-9b89-16bcf3231d87-kube-api-access-mnh7l\") pod \"test-curl-7486-fa2888\" (UID: \"f22a6833-bf52-4c84-9b89-16bcf3231d87\") " pod="opendatahub/test-curl-7486-fa2888" Oct 30 01:52:04.724811 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.724771 5055 generic.go:356] "Generic (PLEG): container finished" podID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" containerID="6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6" exitCode=0 Oct 30 01:52:04.725010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.724823 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" Oct 30 01:52:04.725010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.724857 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" event={"ID":"ee3e48ad-a4ca-418f-843e-46d87af6ee94","Type":"ContainerDied","Data":"6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6"} Oct 30 01:52:04.725010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.724938 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl" event={"ID":"ee3e48ad-a4ca-418f-843e-46d87af6ee94","Type":"ContainerDied","Data":"7fbab3e3335069009b35662f76fc6c50ba5e9ed5dbbe8cdb02d940b2fe9dd364"} Oct 30 01:52:04.725010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.724966 5055 scope.go:122] "RemoveContainer" containerID="6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6" Oct 30 01:52:04.726678 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.726654 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" event={"ID":"ea0b9039-8705-4ba7-8942-991b5fd80d9b","Type":"ContainerDied","Data":"5ddcf1710a3ff60b2b479e62afa69ac7ed996aae832d1c892475ea2aec829ade"} Oct 30 01:52:04.726804 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.726695 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd" Oct 30 01:52:04.726804 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.726712 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn" Oct 30 01:52:04.737389 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.737280 5055 scope.go:122] "RemoveContainer" containerID="6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6" Oct 30 01:52:04.737786 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:04.737764 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6\": container with ID starting with 6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6 not found: ID does not exist" containerID="6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6" Oct 30 01:52:04.737859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.737797 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6"} err="failed to get container status \"6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6\": rpc error: code = NotFound desc = could not find container \"6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6\": container with ID starting with 6451d007d84c99fb5c67cb1424e8b355fbf6bae2060fff1547adb9873ee86da6 not found: ID does not exist" Oct 30 01:52:04.737859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.737819 5055 scope.go:122] "RemoveContainer" containerID="b3f3fc8af7c7ebfb7b61426f2dc53a146dfc1454907dd67bacf27490a146ee83" Oct 30 01:52:04.750192 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.750165 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-fa2888" Oct 30 01:52:04.808993 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.808895 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd"] Oct 30 01:52:04.827448 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.827421 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beafde80-c491-4977-8a5e-8abe3827fb07" path="/var/lib/kubelet/pods/beafde80-c491-4977-8a5e-8abe3827fb07/volumes" Oct 30 01:52:04.827781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.827764 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-x8ftd"] Oct 30 01:52:04.854753 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.854712 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl"] Oct 30 01:52:04.858380 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.858350 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-pdtbl"] Oct 30 01:52:04.903369 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.900410 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn"] Oct 30 01:52:04.906413 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:04.905394 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-9rvmn"] Oct 30 01:52:05.066223 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:52:05.066188 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf22a6833_bf52_4c84_9b89_16bcf3231d87.slice/crio-809c25c81c1647336f5fce320ce9eba48563c0b6bc34f218ba715462af698d48 WatchSource:0}: Error finding container 809c25c81c1647336f5fce320ce9eba48563c0b6bc34f218ba715462af698d48: Status 404 returned error can't find the container with id 809c25c81c1647336f5fce320ce9eba48563c0b6bc34f218ba715462af698d48 Oct 30 01:52:05.067018 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:05.066941 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-fa2888"] Oct 30 01:52:05.681023 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:05.680993 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 01:52:05.737837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:05.737796 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-fa2888" event={"ID":"f22a6833-bf52-4c84-9b89-16bcf3231d87","Type":"ContainerStarted","Data":"32e57a1069085ae711a05f120bf3d3af7706cbcf5fc604c21e08b1d27c43c605"} Oct 30 01:52:05.738093 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:05.737871 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-fa2888" event={"ID":"f22a6833-bf52-4c84-9b89-16bcf3231d87","Type":"ContainerStarted","Data":"809c25c81c1647336f5fce320ce9eba48563c0b6bc34f218ba715462af698d48"} Oct 30 01:52:05.791517 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:05.779164 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7486-fa2888" podStartSLOduration=1.779143183 podStartE2EDuration="1.779143183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:05.774434695 +0000 UTC m=+2093.095391735" watchObservedRunningTime="2026-10-30 01:52:05.779143183 +0000 UTC m=+2093.100100225" Oct 30 01:52:06.490027 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.489991 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph"] Oct 30 01:52:06.490297 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.490272 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" containerID="cri-o://364d65fc17f9484354bc685744f3deb975419bdad10a1607eae1d5fff4b89192" gracePeriod=30 Oct 30 01:52:06.502691 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.502653 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t"] Oct 30 01:52:06.502939 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.502911 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerName="istio-proxy" containerID="cri-o://e836f07493e78d9562c862fca1fda8c46b525612268a93aee9caf9548c0e4d72" gracePeriod=30 Oct 30 01:52:06.506207 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.506169 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78"] Oct 30 01:52:06.507009 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.506587 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" containerID="cri-o://6bbf4d5d00f532c448667b4b8b135b5769ab6af83472e4d5bbffcc2f27cb4bfa" gracePeriod=30 Oct 30 01:52:06.637821 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.637782 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7486-fa2888"] Oct 30 01:52:06.643670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.643635 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7486-fa2888"] Oct 30 01:52:06.825705 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.825610 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcfcf547-3b5f-4907-b88e-392969c79158" path="/var/lib/kubelet/pods/dcfcf547-3b5f-4907-b88e-392969c79158/volumes" Oct 30 01:52:06.826336 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.826307 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea0b9039-8705-4ba7-8942-991b5fd80d9b" path="/var/lib/kubelet/pods/ea0b9039-8705-4ba7-8942-991b5fd80d9b/volumes" Oct 30 01:52:06.826904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.826886 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3e48ad-a4ca-418f-843e-46d87af6ee94" path="/var/lib/kubelet/pods/ee3e48ad-a4ca-418f-843e-46d87af6ee94/volumes" Oct 30 01:52:06.827818 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.827797 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7486-2c0a8a"] Oct 30 01:52:06.828509 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.828493 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f22a6833-bf52-4c84-9b89-16bcf3231d87" containerName="test-curl-7486-fa2888" Oct 30 01:52:06.828509 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.828509 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f22a6833-bf52-4c84-9b89-16bcf3231d87" containerName="test-curl-7486-fa2888" Oct 30 01:52:06.828651 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.828641 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f22a6833-bf52-4c84-9b89-16bcf3231d87" containerName="test-curl-7486-fa2888" Oct 30 01:52:06.833434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.833414 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-2c0a8a" Oct 30 01:52:06.834036 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.834016 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-2c0a8a"] Oct 30 01:52:06.977298 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:06.977253 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7v9s\" (UniqueName: \"kubernetes.io/projected/d30c9072-404d-4bb1-84a8-45ac933a1e83-kube-api-access-g7v9s\") pod \"test-curl-7486-2c0a8a\" (UID: \"d30c9072-404d-4bb1-84a8-45ac933a1e83\") " pod="opendatahub/test-curl-7486-2c0a8a" Oct 30 01:52:07.079904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:07.079806 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7v9s\" (UniqueName: \"kubernetes.io/projected/d30c9072-404d-4bb1-84a8-45ac933a1e83-kube-api-access-g7v9s\") pod \"test-curl-7486-2c0a8a\" (UID: \"d30c9072-404d-4bb1-84a8-45ac933a1e83\") " pod="opendatahub/test-curl-7486-2c0a8a" Oct 30 01:52:07.109747 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:07.109715 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7v9s\" (UniqueName: \"kubernetes.io/projected/d30c9072-404d-4bb1-84a8-45ac933a1e83-kube-api-access-g7v9s\") pod \"test-curl-7486-2c0a8a\" (UID: \"d30c9072-404d-4bb1-84a8-45ac933a1e83\") " pod="opendatahub/test-curl-7486-2c0a8a" Oct 30 01:52:07.150470 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:07.150430 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-2c0a8a" Oct 30 01:52:07.517997 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:52:07.517962 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd30c9072_404d_4bb1_84a8_45ac933a1e83.slice/crio-ae2279b957e0cc2bda1b28f03cadf2bd62ac5fa42778cc671f043190bbbee2ea WatchSource:0}: Error finding container ae2279b957e0cc2bda1b28f03cadf2bd62ac5fa42778cc671f043190bbbee2ea: Status 404 returned error can't find the container with id ae2279b957e0cc2bda1b28f03cadf2bd62ac5fa42778cc671f043190bbbee2ea Oct 30 01:52:07.519367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:07.519317 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-2c0a8a"] Oct 30 01:52:07.756928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:07.756886 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-2c0a8a" event={"ID":"d30c9072-404d-4bb1-84a8-45ac933a1e83","Type":"ContainerStarted","Data":"1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3"} Oct 30 01:52:07.756928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:07.756936 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-2c0a8a" event={"ID":"d30c9072-404d-4bb1-84a8-45ac933a1e83","Type":"ContainerStarted","Data":"ae2279b957e0cc2bda1b28f03cadf2bd62ac5fa42778cc671f043190bbbee2ea"} Oct 30 01:52:07.757161 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:07.757149 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7486-fa2888" podUID="f22a6833-bf52-4c84-9b89-16bcf3231d87" containerName="test-curl-7486-fa2888" containerID="cri-o://32e57a1069085ae711a05f120bf3d3af7706cbcf5fc604c21e08b1d27c43c605" gracePeriod=2 Oct 30 01:52:07.796339 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:07.796211 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7486-2c0a8a" podStartSLOduration=1.796196175 podStartE2EDuration="1.796196175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:07.793758161 +0000 UTC m=+2095.114715198" watchObservedRunningTime="2026-10-30 01:52:07.796196175 +0000 UTC m=+2095.117153214" Oct 30 01:52:08.612923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.612247 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7486-2c0a8a"] Oct 30 01:52:08.615395 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.615361 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7486-2c0a8a"] Oct 30 01:52:08.775347 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.775279 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7486-60c7fa"] Oct 30 01:52:08.778596 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.778283 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d30c9072-404d-4bb1-84a8-45ac933a1e83" containerName="test-curl-7486-2c0a8a" Oct 30 01:52:08.778596 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.778313 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30c9072-404d-4bb1-84a8-45ac933a1e83" containerName="test-curl-7486-2c0a8a" Oct 30 01:52:08.779414 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.778878 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30c9072-404d-4bb1-84a8-45ac933a1e83" containerName="test-curl-7486-2c0a8a" Oct 30 01:52:08.785904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.785544 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-60c7fa"] Oct 30 01:52:08.785904 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.785658 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-60c7fa" Oct 30 01:52:08.828816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.828773 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtq5j\" (UniqueName: \"kubernetes.io/projected/96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a-kube-api-access-wtq5j\") pod \"test-curl-7486-60c7fa\" (UID: \"96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a\") " pod="opendatahub/test-curl-7486-60c7fa" Oct 30 01:52:08.929867 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.929771 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtq5j\" (UniqueName: \"kubernetes.io/projected/96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a-kube-api-access-wtq5j\") pod \"test-curl-7486-60c7fa\" (UID: \"96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a\") " pod="opendatahub/test-curl-7486-60c7fa" Oct 30 01:52:08.949985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:08.949954 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtq5j\" (UniqueName: \"kubernetes.io/projected/96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a-kube-api-access-wtq5j\") pod \"test-curl-7486-60c7fa\" (UID: \"96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a\") " pod="opendatahub/test-curl-7486-60c7fa" Oct 30 01:52:09.109547 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:09.109513 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-60c7fa" Oct 30 01:52:09.418194 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:52:09.418160 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96b81cc6_196a_4c79_9ac7_a2b0c4d67a6a.slice/crio-4709b5ffd4ad4b326afffdf4a32f11813d4ba93258af00212d27c921bc49e301 WatchSource:0}: Error finding container 4709b5ffd4ad4b326afffdf4a32f11813d4ba93258af00212d27c921bc49e301: Status 404 returned error can't find the container with id 4709b5ffd4ad4b326afffdf4a32f11813d4ba93258af00212d27c921bc49e301 Oct 30 01:52:09.420149 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:09.420119 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-60c7fa"] Oct 30 01:52:09.787450 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:09.787414 5055 generic.go:356] "Generic (PLEG): container finished" podID="f22a6833-bf52-4c84-9b89-16bcf3231d87" containerID="32e57a1069085ae711a05f120bf3d3af7706cbcf5fc604c21e08b1d27c43c605" exitCode=137 Oct 30 01:52:09.789696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:09.789518 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-60c7fa" event={"ID":"96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a","Type":"ContainerStarted","Data":"9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a"} Oct 30 01:52:09.789696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:09.789571 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-60c7fa" event={"ID":"96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a","Type":"ContainerStarted","Data":"4709b5ffd4ad4b326afffdf4a32f11813d4ba93258af00212d27c921bc49e301"} Oct 30 01:52:09.789696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:09.789565 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7486-2c0a8a" podUID="d30c9072-404d-4bb1-84a8-45ac933a1e83" containerName="test-curl-7486-2c0a8a" containerID="cri-o://1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3" gracePeriod=2 Oct 30 01:52:09.827945 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:09.827891 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7486-60c7fa" podStartSLOduration=1.827876142 podStartE2EDuration="1.827876142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:09.826843898 +0000 UTC m=+2097.147800938" watchObservedRunningTime="2026-10-30 01:52:09.827876142 +0000 UTC m=+2097.148833182" Oct 30 01:52:09.941568 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:09.941530 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-fa2888" Oct 30 01:52:10.046216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:10.046125 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f22a6833-bf52-4c84-9b89-16bcf3231d87-kube-api-access-mnh7l\" (UniqueName: \"kubernetes.io/projected/f22a6833-bf52-4c84-9b89-16bcf3231d87-kube-api-access-mnh7l\") pod \"f22a6833-bf52-4c84-9b89-16bcf3231d87\" (UID: \"f22a6833-bf52-4c84-9b89-16bcf3231d87\") " Oct 30 01:52:10.048570 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:10.048540 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22a6833-bf52-4c84-9b89-16bcf3231d87-kube-api-access-mnh7l" pod "f22a6833-bf52-4c84-9b89-16bcf3231d87" (UID: "f22a6833-bf52-4c84-9b89-16bcf3231d87"). InnerVolumeSpecName "kube-api-access-mnh7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:10.148308 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:10.148275 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnh7l\" (UniqueName: \"kubernetes.io/projected/f22a6833-bf52-4c84-9b89-16bcf3231d87-kube-api-access-mnh7l\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:10.645527 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:10.645481 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7486-60c7fa"] Oct 30 01:52:10.649724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:10.649689 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7486-60c7fa"] Oct 30 01:52:10.798449 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:10.798413 5055 scope.go:122] "RemoveContainer" containerID="32e57a1069085ae711a05f120bf3d3af7706cbcf5fc604c21e08b1d27c43c605" Oct 30 01:52:10.798927 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:10.798428 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-fa2888" Oct 30 01:52:10.820302 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:10.820275 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f22a6833-bf52-4c84-9b89-16bcf3231d87" path="/var/lib/kubelet/pods/f22a6833-bf52-4c84-9b89-16bcf3231d87/volumes" Oct 30 01:52:10.825150 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:10.825118 5055 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7486-fa2888\" is forbidden: User \"system:node:ip-10-0-134-254.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-254.ec2.internal' and this object" podUID="f22a6833-bf52-4c84-9b89-16bcf3231d87" pod="opendatahub/test-curl-7486-fa2888" Oct 30 01:52:11.821795 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.821763 5055 generic.go:356] "Generic (PLEG): container finished" podID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerID="364d65fc17f9484354bc685744f3deb975419bdad10a1607eae1d5fff4b89192" exitCode=0 Oct 30 01:52:11.822278 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.821875 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" event={"ID":"a76261f8-01b6-4764-be6b-cd36da5356ad","Type":"ContainerDied","Data":"364d65fc17f9484354bc685744f3deb975419bdad10a1607eae1d5fff4b89192"} Oct 30 01:52:11.825629 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.825600 5055 generic.go:356] "Generic (PLEG): container finished" podID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerID="e836f07493e78d9562c862fca1fda8c46b525612268a93aee9caf9548c0e4d72" exitCode=0 Oct 30 01:52:11.825956 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.825838 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" event={"ID":"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94","Type":"ContainerDied","Data":"e836f07493e78d9562c862fca1fda8c46b525612268a93aee9caf9548c0e4d72"} Oct 30 01:52:11.828267 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.828241 5055 generic.go:356] "Generic (PLEG): container finished" podID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerID="6bbf4d5d00f532c448667b4b8b135b5769ab6af83472e4d5bbffcc2f27cb4bfa" exitCode=0 Oct 30 01:52:11.828407 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.828318 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" event={"ID":"8186c325-cf8c-4a7b-b0ac-e92058eb18ce","Type":"ContainerDied","Data":"6bbf4d5d00f532c448667b4b8b135b5769ab6af83472e4d5bbffcc2f27cb4bfa"} Oct 30 01:52:11.830773 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.830745 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7486-60c7fa" podUID="96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a" containerName="test-curl-7486-60c7fa" containerID="cri-o://9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a" gracePeriod=2 Oct 30 01:52:11.912353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.912070 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:52:11.917799 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.917770 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:52:11.922983 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.922956 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:52:11.979423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979376 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-podinfo\") pod \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " Oct 30 01:52:11.979588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979428 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-certs\") pod \"a76261f8-01b6-4764-be6b-cd36da5356ad\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " Oct 30 01:52:11.979588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979461 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-podinfo\") pod \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " Oct 30 01:52:11.979588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979509 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-kube-api-access-zntsf\" (UniqueName: \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-kube-api-access-zntsf\") pod \"a76261f8-01b6-4764-be6b-cd36da5356ad\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " Oct 30 01:52:11.979588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979537 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-envoy\") pod \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " Oct 30 01:52:11.979588 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979570 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-token\" (UniqueName: \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-token\") pod \"a76261f8-01b6-4764-be6b-cd36da5356ad\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " Oct 30 01:52:11.979810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979627 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-socket\") pod \"a76261f8-01b6-4764-be6b-cd36da5356ad\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " Oct 30 01:52:11.979810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979683 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-token\" (UniqueName: \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-token\") pod \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " Oct 30 01:52:11.979810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979738 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istiod-ca-cert\") pod \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " Oct 30 01:52:11.979810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979768 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-socket\") pod \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " Oct 30 01:52:11.979810 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979799 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-kube-api-access-68rw2\" (UniqueName: \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-kube-api-access-68rw2\") pod \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " Oct 30 01:52:11.980034 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979832 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-data\") pod \"a76261f8-01b6-4764-be6b-cd36da5356ad\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " Oct 30 01:52:11.980034 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979863 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-socket\") pod \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " Oct 30 01:52:11.980034 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979907 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-envoy\") pod \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " Oct 30 01:52:11.980034 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979934 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istiod-ca-cert\") pod \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " Oct 30 01:52:11.980034 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.979976 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-kube-api-access-ptr8l\" (UniqueName: \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-kube-api-access-ptr8l\") pod \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " Oct 30 01:52:11.980616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980082 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-credential-socket\") pod \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " Oct 30 01:52:11.980616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980111 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-certs\") pod \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " Oct 30 01:52:11.980616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980146 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-data\") pod \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\" (UID: \"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94\") " Oct 30 01:52:11.980616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980210 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-podinfo\") pod \"a76261f8-01b6-4764-be6b-cd36da5356ad\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " Oct 30 01:52:11.980616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980259 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-credential-socket\") pod \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " Oct 30 01:52:11.980616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980380 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a76261f8-01b6-4764-be6b-cd36da5356ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76261f8-01b6-4764-be6b-cd36da5356ad-istiod-ca-cert\") pod \"a76261f8-01b6-4764-be6b-cd36da5356ad\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " Oct 30 01:52:11.980616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980431 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-certs\") pod \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " Oct 30 01:52:11.980616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980495 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-envoy\") pod \"a76261f8-01b6-4764-be6b-cd36da5356ad\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " Oct 30 01:52:11.980616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980549 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-credential-socket\") pod \"a76261f8-01b6-4764-be6b-cd36da5356ad\" (UID: \"a76261f8-01b6-4764-be6b-cd36da5356ad\") " Oct 30 01:52:11.980616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980595 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-token\" (UniqueName: \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-token\") pod \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " Oct 30 01:52:11.980952 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.980625 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-data\") pod \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\" (UID: \"8186c325-cf8c-4a7b-b0ac-e92058eb18ce\") " Oct 30 01:52:11.981493 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.981254 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-socket" pod "8186c325-cf8c-4a7b-b0ac-e92058eb18ce" (UID: "8186c325-cf8c-4a7b-b0ac-e92058eb18ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.981722 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.981657 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-socket" pod "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" (UID: "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.983793 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.981906 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-socket" pod "a76261f8-01b6-4764-be6b-cd36da5356ad" (UID: "a76261f8-01b6-4764-be6b-cd36da5356ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.983793 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.982120 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istiod-ca-cert" pod "8186c325-cf8c-4a7b-b0ac-e92058eb18ce" (UID: "8186c325-cf8c-4a7b-b0ac-e92058eb18ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:11.983793 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.982364 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-credential-socket" pod "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" (UID: "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.983793 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.982714 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istiod-ca-cert" pod "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" (UID: "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:11.984552 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.984519 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-credential-socket" pod "8186c325-cf8c-4a7b-b0ac-e92058eb18ce" (UID: "8186c325-cf8c-4a7b-b0ac-e92058eb18ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.984842 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.984824 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-certs" pod "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" (UID: "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.988928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.985799 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-credential-socket" pod "a76261f8-01b6-4764-be6b-cd36da5356ad" (UID: "a76261f8-01b6-4764-be6b-cd36da5356ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.989087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.986100 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a76261f8-01b6-4764-be6b-cd36da5356ad-istiod-ca-cert" pod "a76261f8-01b6-4764-be6b-cd36da5356ad" (UID: "a76261f8-01b6-4764-be6b-cd36da5356ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:11.989214 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.986287 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-certs" pod "8186c325-cf8c-4a7b-b0ac-e92058eb18ce" (UID: "8186c325-cf8c-4a7b-b0ac-e92058eb18ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.989421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.986431 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-data" pod "8186c325-cf8c-4a7b-b0ac-e92058eb18ce" (UID: "8186c325-cf8c-4a7b-b0ac-e92058eb18ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.989421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.986557 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-data" pod "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" (UID: "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.989421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.987883 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-data" pod "a76261f8-01b6-4764-be6b-cd36da5356ad" (UID: "a76261f8-01b6-4764-be6b-cd36da5356ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.989421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.988062 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-certs" pod "a76261f8-01b6-4764-be6b-cd36da5356ad" (UID: "a76261f8-01b6-4764-be6b-cd36da5356ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.989839 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.989821 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-token" pod "a76261f8-01b6-4764-be6b-cd36da5356ad" (UID: "a76261f8-01b6-4764-be6b-cd36da5356ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:11.991660 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.991640 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-envoy" pod "a76261f8-01b6-4764-be6b-cd36da5356ad" (UID: "a76261f8-01b6-4764-be6b-cd36da5356ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.993528 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.993486 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-podinfo" pod "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" (UID: "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:11.994004 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.993980 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-envoy" pod "8186c325-cf8c-4a7b-b0ac-e92058eb18ce" (UID: "8186c325-cf8c-4a7b-b0ac-e92058eb18ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:11.994191 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.994173 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-kube-api-access-zntsf" pod "a76261f8-01b6-4764-be6b-cd36da5356ad" (UID: "a76261f8-01b6-4764-be6b-cd36da5356ad"). InnerVolumeSpecName "kube-api-access-zntsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:11.994289 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.994273 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-token" pod "8186c325-cf8c-4a7b-b0ac-e92058eb18ce" (UID: "8186c325-cf8c-4a7b-b0ac-e92058eb18ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:11.994425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.994402 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-podinfo" pod "8186c325-cf8c-4a7b-b0ac-e92058eb18ce" (UID: "8186c325-cf8c-4a7b-b0ac-e92058eb18ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:11.995427 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.995403 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-podinfo" pod "a76261f8-01b6-4764-be6b-cd36da5356ad" (UID: "a76261f8-01b6-4764-be6b-cd36da5356ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:11.997078 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:11.997052 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-kube-api-access-ptr8l" pod "8186c325-cf8c-4a7b-b0ac-e92058eb18ce" (UID: "8186c325-cf8c-4a7b-b0ac-e92058eb18ce"). InnerVolumeSpecName "kube-api-access-ptr8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:12.010426 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.010318 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-token" pod "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" (UID: "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:12.010729 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.010693 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-envoy" pod "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" (UID: "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:12.013134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.013097 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-kube-api-access-68rw2" pod "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" (UID: "7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94"). InnerVolumeSpecName "kube-api-access-68rw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:12.048853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.048774 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-2c0a8a" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.082166 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30c9072-404d-4bb1-84a8-45ac933a1e83-kube-api-access-g7v9s\" (UniqueName: \"kubernetes.io/projected/d30c9072-404d-4bb1-84a8-45ac933a1e83-kube-api-access-g7v9s\") pod \"d30c9072-404d-4bb1-84a8-45ac933a1e83\" (UID: \"d30c9072-404d-4bb1-84a8-45ac933a1e83\") " Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.082866 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.082890 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.082908 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.082924 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.082938 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.082953 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zntsf\" (UniqueName: \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-kube-api-access-zntsf\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.082968 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.082982 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.082996 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083011 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083027 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083044 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083059 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68rw2\" (UniqueName: \"kubernetes.io/projected/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-kube-api-access-68rw2\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083073 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083088 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083104 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083118 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083133 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptr8l\" (UniqueName: \"kubernetes.io/projected/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-kube-api-access-ptr8l\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083146 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083162 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083176 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083188 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083204 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083219 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a76261f8-01b6-4764-be6b-cd36da5356ad-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083235 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8186c325-cf8c-4a7b-b0ac-e92058eb18ce-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083249 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.083735 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.083264 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a76261f8-01b6-4764-be6b-cd36da5356ad-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.088888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.088832 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30c9072-404d-4bb1-84a8-45ac933a1e83-kube-api-access-g7v9s" pod "d30c9072-404d-4bb1-84a8-45ac933a1e83" (UID: "d30c9072-404d-4bb1-84a8-45ac933a1e83"). InnerVolumeSpecName "kube-api-access-g7v9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:12.184743 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.184708 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7v9s\" (UniqueName: \"kubernetes.io/projected/d30c9072-404d-4bb1-84a8-45ac933a1e83-kube-api-access-g7v9s\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:12.636077 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.636039 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k"] Oct 30 01:52:12.636890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.636864 5055 cpu_manager.go:414] "containerMap: removing container" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" Oct 30 01:52:12.636890 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.636888 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" Oct 30 01:52:12.637121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.636919 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerName="istio-proxy" Oct 30 01:52:12.637121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.636928 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerName="istio-proxy" Oct 30 01:52:12.637121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.636960 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" Oct 30 01:52:12.637121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.636967 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" Oct 30 01:52:12.637121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.636996 5055 cpu_manager.go:414] "containerMap: removing container" podUID="96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a" containerName="test-curl-7486-60c7fa" Oct 30 01:52:12.637121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.637004 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a" containerName="test-curl-7486-60c7fa" Oct 30 01:52:12.637627 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.637172 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" containerName="istio-proxy" Oct 30 01:52:12.637627 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.637191 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" containerName="istio-proxy" Oct 30 01:52:12.637627 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.637208 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" containerName="istio-proxy" Oct 30 01:52:12.637627 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.637256 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a" containerName="test-curl-7486-60c7fa" Oct 30 01:52:12.641707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.641660 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.664016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.662219 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-76a19e-openshift-default-dockercfg-8g67k\"" Oct 30 01:52:12.701367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.692399 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-socket\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.701367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.692466 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6rzb\" (UniqueName: \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-kube-api-access-z6rzb\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.701367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.692550 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-envoy\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.701367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.692644 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-podinfo\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.701367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.692707 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79831cb6-7cc2-4153-b501-079ef3bbe670-istiod-ca-cert\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.701367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.692770 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-token\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.701367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.692815 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-credential-socket\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.701367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.692848 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-data\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.701367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.692955 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-certs\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.733538 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.733502 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k"] Oct 30 01:52:12.815710 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.815612 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-certs\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.816013 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.815992 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-socket\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.816141 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.816125 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rzb\" (UniqueName: \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-kube-api-access-z6rzb\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.816297 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.816280 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-envoy\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.816519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.816503 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-podinfo\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.816659 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.816645 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79831cb6-7cc2-4153-b501-079ef3bbe670-istiod-ca-cert\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.816807 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.816793 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-token\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.816928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.816914 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-credential-socket\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.817035 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.817021 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-data\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.823833 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.823797 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-credential-socket\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.859421 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.824974 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-data\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.859625 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.832693 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79831cb6-7cc2-4153-b501-079ef3bbe670-istiod-ca-cert\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.859701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.839287 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-socket\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.859763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.839846 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-certs\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.859823 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.848973 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-envoy\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.860723 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.860697 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" Oct 30 01:52:12.863363 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.863340 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" Oct 30 01:52:12.865647 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.864178 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30c9072-404d-4bb1-84a8-45ac933a1e83" path="/var/lib/kubelet/pods/d30c9072-404d-4bb1-84a8-45ac933a1e83/volumes" Oct 30 01:52:12.866280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.865251 5055 generic.go:356] "Generic (PLEG): container finished" podID="d30c9072-404d-4bb1-84a8-45ac933a1e83" containerID="1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3" exitCode=137 Oct 30 01:52:12.866600 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.865345 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-2c0a8a" Oct 30 01:52:12.869953 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.869056 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" Oct 30 01:52:12.870208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.869629 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-podinfo\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.870208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.869688 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-token\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.890349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.886978 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rzb\" (UniqueName: \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-kube-api-access-z6rzb\") pod \"e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:12.946134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.915874 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78" event={"ID":"8186c325-cf8c-4a7b-b0ac-e92058eb18ce","Type":"ContainerDied","Data":"6a799d5810f94a1af2dbc22ed77a8f877b5a1d679caf66ba062588e49662789e"} Oct 30 01:52:12.946134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.915952 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph" event={"ID":"a76261f8-01b6-4764-be6b-cd36da5356ad","Type":"ContainerDied","Data":"384f19ba9e2a6dc40fa1c12de40ee44ee25a73cf91af45844e391461fe11d59c"} Oct 30 01:52:12.946134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.915985 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t" event={"ID":"7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94","Type":"ContainerDied","Data":"06e9ba175e62c5bf4be74c1a1336ca52c77b43d3f1a8e54d46f0ceb091f1faed"} Oct 30 01:52:12.946134 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.923429 5055 scope.go:122] "RemoveContainer" containerID="6bbf4d5d00f532c448667b4b8b135b5769ab6af83472e4d5bbffcc2f27cb4bfa" Oct 30 01:52:12.984492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.984360 5055 scope.go:122] "RemoveContainer" containerID="364d65fc17f9484354bc685744f3deb975419bdad10a1607eae1d5fff4b89192" Oct 30 01:52:12.984858 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.984823 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-76a19e-openshift-default-dockercfg-8g67k\"" Oct 30 01:52:12.986159 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:12.985941 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:13.034795 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.034762 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph"] Oct 30 01:52:13.044803 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.044773 5055 scope.go:122] "RemoveContainer" containerID="1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3" Oct 30 01:52:13.047067 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.047029 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-srqph"] Oct 30 01:52:13.057615 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.056978 5055 scope.go:122] "RemoveContainer" containerID="1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3" Oct 30 01:52:13.057772 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:13.057731 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3\": container with ID starting with 1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3 not found: ID does not exist" containerID="1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3" Oct 30 01:52:13.057950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.057920 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3"} err="failed to get container status \"1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3\": rpc error: code = NotFound desc = could not find container \"1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3\": container with ID starting with 1269b35d0eedd0b95a921825b6f1b58662d2c5dd55a1531c28ae3d4a424b7ee3 not found: ID does not exist" Oct 30 01:52:13.058041 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.057948 5055 scope.go:122] "RemoveContainer" containerID="e836f07493e78d9562c862fca1fda8c46b525612268a93aee9caf9548c0e4d72" Oct 30 01:52:13.078675 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.078642 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t"] Oct 30 01:52:13.087093 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.087040 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-n667t"] Oct 30 01:52:13.123153 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.123113 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78"] Oct 30 01:52:13.126973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.126941 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-npz78"] Oct 30 01:52:13.239519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.239426 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:52:13.249910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.249884 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:52:13.352095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.351959 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k"] Oct 30 01:52:13.354723 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.354689 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:52:13.354839 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.354764 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:52:13.354839 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.354795 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:52:13.883977 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.883941 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" event={"ID":"79831cb6-7cc2-4153-b501-079ef3bbe670","Type":"ContainerStarted","Data":"d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549"} Oct 30 01:52:13.884594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.884006 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" event={"ID":"79831cb6-7cc2-4153-b501-079ef3bbe670","Type":"ContainerStarted","Data":"71fd595f51960577aa123a2ed184c54fbfaf2847f73005ff69fbfb9b2f116d3f"} Oct 30 01:52:13.928198 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.928144 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" podStartSLOduration=1.928122713 podStartE2EDuration="1.928122713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:13.925505788 +0000 UTC m=+2101.246462829" watchObservedRunningTime="2026-10-30 01:52:13.928122713 +0000 UTC m=+2101.249079753" Oct 30 01:52:13.988718 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:13.988632 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:14.137287 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.137260 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-60c7fa" Oct 30 01:52:14.240112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.240076 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a-kube-api-access-wtq5j\" (UniqueName: \"kubernetes.io/projected/96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a-kube-api-access-wtq5j\") pod \"96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a\" (UID: \"96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a\") " Oct 30 01:52:14.242473 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.242442 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a-kube-api-access-wtq5j" pod "96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a" (UID: "96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a"). InnerVolumeSpecName "kube-api-access-wtq5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:14.341704 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.341666 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtq5j\" (UniqueName: \"kubernetes.io/projected/96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a-kube-api-access-wtq5j\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:14.452425 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.452313 5055 scope.go:122] "RemoveContainer" containerID="d4c68c868f28f4b9ea4e54a20bb39b80244e3faf8ed35345b61f37b4f1f58090" Oct 30 01:52:14.538820 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:14.536103 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:52:14.846240 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.846199 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94" path="/var/lib/kubelet/pods/7ed9cbd9-d3ee-4100-a74b-cdbfd6165e94/volumes" Oct 30 01:52:14.846841 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.846819 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8186c325-cf8c-4a7b-b0ac-e92058eb18ce" path="/var/lib/kubelet/pods/8186c325-cf8c-4a7b-b0ac-e92058eb18ce/volumes" Oct 30 01:52:14.847230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.847213 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a" path="/var/lib/kubelet/pods/96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a/volumes" Oct 30 01:52:14.848010 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.847990 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a76261f8-01b6-4764-be6b-cd36da5356ad" path="/var/lib/kubelet/pods/a76261f8-01b6-4764-be6b-cd36da5356ad/volumes" Oct 30 01:52:14.899463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.899430 5055 generic.go:356] "Generic (PLEG): container finished" podID="96b81cc6-196a-4c79-9ac7-a2b0c4d67a6a" containerID="9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a" exitCode=137 Oct 30 01:52:14.899945 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.899591 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-60c7fa" Oct 30 01:52:14.899945 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.899657 5055 scope.go:122] "RemoveContainer" containerID="9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a" Oct 30 01:52:14.918185 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.918161 5055 scope.go:122] "RemoveContainer" containerID="9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a" Oct 30 01:52:14.918545 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:14.918514 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a\": container with ID starting with 9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a not found: ID does not exist" containerID="9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a" Oct 30 01:52:14.918678 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.918553 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a"} err="failed to get container status \"9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a\": rpc error: code = NotFound desc = could not find container \"9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a\": container with ID starting with 9f6007b596a62f21cd4913e7ed5c215a7f2b373250483fd24272f5077e4ebc8a not found: ID does not exist" Oct 30 01:52:14.993806 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:14.993775 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:15.909917 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:15.909887 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:15.911019 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:15.911000 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:52:17.112033 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:17.111994 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:52:17.940055 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:17.940019 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk"] Oct 30 01:52:17.947629 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:17.947592 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 01:52:17.964539 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:17.964507 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk"] Oct 30 01:52:17.973294 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:17.973165 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-76a19e\"" Oct 30 01:52:18.012206 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.012158 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw"] Oct 30 01:52:18.017349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.017299 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 01:52:18.026898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.026859 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8cmj\" (UniqueName: \"kubernetes.io/projected/65f34b67-7d82-405f-8825-8fede2564631-kube-api-access-f8cmj\") pod \"payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk\" (UID: \"65f34b67-7d82-405f-8825-8fede2564631\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 01:52:18.027110 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.027085 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f34b67-7d82-405f-8825-8fede2564631-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk\" (UID: \"65f34b67-7d82-405f-8825-8fede2564631\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 01:52:18.037748 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.037704 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-76a19e-dockercfg-bn9sj\"" Oct 30 01:52:18.044129 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.044097 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw"] Oct 30 01:52:18.145370 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.145305 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae14669-6647-4386-a4fd-7a769705984e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw\" (UID: \"1ae14669-6647-4386-a4fd-7a769705984e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 01:52:18.145894 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.145435 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8cmj\" (UniqueName: \"kubernetes.io/projected/65f34b67-7d82-405f-8825-8fede2564631-kube-api-access-f8cmj\") pod \"payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk\" (UID: \"65f34b67-7d82-405f-8825-8fede2564631\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 01:52:18.145894 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.145479 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfsrn\" (UniqueName: \"kubernetes.io/projected/1ae14669-6647-4386-a4fd-7a769705984e-kube-api-access-wfsrn\") pod \"payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw\" (UID: \"1ae14669-6647-4386-a4fd-7a769705984e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 01:52:18.145894 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.145592 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f34b67-7d82-405f-8825-8fede2564631-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk\" (UID: \"65f34b67-7d82-405f-8825-8fede2564631\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 01:52:18.179401 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.146623 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f34b67-7d82-405f-8825-8fede2564631-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk\" (UID: \"65f34b67-7d82-405f-8825-8fede2564631\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 01:52:18.198405 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.196038 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8cmj\" (UniqueName: \"kubernetes.io/projected/65f34b67-7d82-405f-8825-8fede2564631-kube-api-access-f8cmj\") pod \"payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk\" (UID: \"65f34b67-7d82-405f-8825-8fede2564631\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 01:52:18.246859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.246825 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae14669-6647-4386-a4fd-7a769705984e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw\" (UID: \"1ae14669-6647-4386-a4fd-7a769705984e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 01:52:18.247060 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.246901 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsrn\" (UniqueName: \"kubernetes.io/projected/1ae14669-6647-4386-a4fd-7a769705984e-kube-api-access-wfsrn\") pod \"payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw\" (UID: \"1ae14669-6647-4386-a4fd-7a769705984e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 01:52:18.247649 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.247627 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae14669-6647-4386-a4fd-7a769705984e-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw\" (UID: \"1ae14669-6647-4386-a4fd-7a769705984e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 01:52:18.264943 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.264898 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 01:52:18.274211 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.274176 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsrn\" (UniqueName: \"kubernetes.io/projected/1ae14669-6647-4386-a4fd-7a769705984e-kube-api-access-wfsrn\") pod \"payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw\" (UID: \"1ae14669-6647-4386-a4fd-7a769705984e\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 01:52:18.333736 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.333700 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 01:52:18.383381 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.362108 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5"] Oct 30 01:52:18.383381 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.362761 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" containerID="cri-o://f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279" gracePeriod=30 Oct 30 01:52:18.580765 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.580724 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk"] Oct 30 01:52:18.583030 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:52:18.583001 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f34b67_7d82_405f_8825_8fede2564631.slice/crio-dc2655e893591d86f5c395f3bca42ca5579acd136b80bb2814f4c2819b10fc96 WatchSource:0}: Error finding container dc2655e893591d86f5c395f3bca42ca5579acd136b80bb2814f4c2819b10fc96: Status 404 returned error can't find the container with id dc2655e893591d86f5c395f3bca42ca5579acd136b80bb2814f4c2819b10fc96 Oct 30 01:52:18.671940 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:52:18.671742 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae14669_6647_4386_a4fd_7a769705984e.slice/crio-64d86e0801379142db61911ed1d207bc94b2db83e7b53b495ada74eff2f2ac76 WatchSource:0}: Error finding container 64d86e0801379142db61911ed1d207bc94b2db83e7b53b495ada74eff2f2ac76: Status 404 returned error can't find the container with id 64d86e0801379142db61911ed1d207bc94b2db83e7b53b495ada74eff2f2ac76 Oct 30 01:52:18.674571 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.674538 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw"] Oct 30 01:52:18.940856 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.940758 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" event={"ID":"65f34b67-7d82-405f-8825-8fede2564631","Type":"ContainerStarted","Data":"ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99"} Oct 30 01:52:18.940856 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.940840 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" event={"ID":"65f34b67-7d82-405f-8825-8fede2564631","Type":"ContainerStarted","Data":"dc2655e893591d86f5c395f3bca42ca5579acd136b80bb2814f4c2819b10fc96"} Oct 30 01:52:18.941099 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.940991 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 01:52:18.942571 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.942545 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" event={"ID":"1ae14669-6647-4386-a4fd-7a769705984e","Type":"ContainerStarted","Data":"061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8"} Oct 30 01:52:18.942703 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.942583 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" event={"ID":"1ae14669-6647-4386-a4fd-7a769705984e","Type":"ContainerStarted","Data":"64d86e0801379142db61911ed1d207bc94b2db83e7b53b495ada74eff2f2ac76"} Oct 30 01:52:18.942703 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.942688 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 01:52:18.983306 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:18.983243 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" podStartSLOduration=1.98322219 podStartE2EDuration="1.98322219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:18.980027869 +0000 UTC m=+2106.300984911" watchObservedRunningTime="2026-10-30 01:52:18.98322219 +0000 UTC m=+2106.304179235" Oct 30 01:52:19.020914 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:19.020848 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" podStartSLOduration=2.020827525 podStartE2EDuration="2.020827525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:19.018733735 +0000 UTC m=+2106.339690775" watchObservedRunningTime="2026-10-30 01:52:19.020827525 +0000 UTC m=+2106.341784566" Oct 30 01:52:19.234680 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:19.234642 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:52:22.539103 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:22.539011 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-c56z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:52:22.539103 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:22.539077 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:52:23.696201 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.696168 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:52:23.827169 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827129 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-credential-socket\") pod \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " Oct 30 01:52:23.827397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827230 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istiod-ca-cert\") pod \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " Oct 30 01:52:23.827397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827301 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-socket\") pod \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " Oct 30 01:52:23.827397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827370 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-kube-api-access-5glgs\" (UniqueName: \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-kube-api-access-5glgs\") pod \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " Oct 30 01:52:23.827397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827393 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-certs\") pod \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " Oct 30 01:52:23.827666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827411 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-token\" (UniqueName: \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-token\") pod \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " Oct 30 01:52:23.827666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827435 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-credential-socket" pod "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" (UID: "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:23.827666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827471 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-podinfo\") pod \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " Oct 30 01:52:23.827666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827592 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-data\") pod \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " Oct 30 01:52:23.827666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827627 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-envoy\") pod \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\" (UID: \"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e\") " Oct 30 01:52:23.827941 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827843 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-certs" pod "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" (UID: "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:23.827941 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827851 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istiod-ca-cert" pod "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" (UID: "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:23.827941 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827855 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-socket" pod "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" (UID: "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:23.828121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.827958 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-data" pod "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" (UID: "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:23.828189 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.828158 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.828189 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.828178 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.828264 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.828190 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.828264 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.828202 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.828264 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.828211 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.830195 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.830172 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-envoy" pod "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" (UID: "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:23.830195 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.830187 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-kube-api-access-5glgs" pod "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" (UID: "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e"). InnerVolumeSpecName "kube-api-access-5glgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:23.830520 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.830497 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-podinfo" pod "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" (UID: "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:23.830520 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.830509 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-token" pod "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" (UID: "2206e13b-cd58-44c1-ae6d-964bbe9b8d5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:23.929873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.929787 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.929873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.929818 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5glgs\" (UniqueName: \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-kube-api-access-5glgs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.929873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.929830 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.929873 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.929840 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:23.983433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.983377 5055 generic.go:356] "Generic (PLEG): container finished" podID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerID="f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279" exitCode=0 Oct 30 01:52:23.983633 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.983435 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" Oct 30 01:52:23.983633 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.983455 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" event={"ID":"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e","Type":"ContainerDied","Data":"f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279"} Oct 30 01:52:23.983633 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.983525 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5" event={"ID":"2206e13b-cd58-44c1-ae6d-964bbe9b8d5e","Type":"ContainerDied","Data":"8a713ced9d689625d87f10076caf94ea5e639038e16a7948380baafa8a55c94d"} Oct 30 01:52:23.983633 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.983557 5055 scope.go:122] "RemoveContainer" containerID="f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279" Oct 30 01:52:23.997887 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.997863 5055 scope.go:122] "RemoveContainer" containerID="f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279" Oct 30 01:52:23.999884 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:23.999320 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279\": container with ID starting with f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279 not found: ID does not exist" containerID="f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279" Oct 30 01:52:23.999884 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:23.999394 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279"} err="failed to get container status \"f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279\": rpc error: code = NotFound desc = could not find container \"f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279\": container with ID starting with f48972e11d86e51e367cd1e83a4801b56bab194893c00bcf282075c74c72e279 not found: ID does not exist" Oct 30 01:52:24.040988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:24.040932 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5"] Oct 30 01:52:24.080213 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:24.079255 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-c56z5"] Oct 30 01:52:24.822859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:24.822830 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" path="/var/lib/kubelet/pods/2206e13b-cd58-44c1-ae6d-964bbe9b8d5e/volumes" Oct 30 01:52:26.919804 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:26.919757 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd"] Oct 30 01:52:26.920476 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:26.920457 5055 cpu_manager.go:414] "containerMap: removing container" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" Oct 30 01:52:26.920476 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:26.920475 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" Oct 30 01:52:26.920693 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:26.920679 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="2206e13b-cd58-44c1-ae6d-964bbe9b8d5e" containerName="istio-proxy" Oct 30 01:52:26.925882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:26.925852 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:26.950435 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:26.950406 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd"] Oct 30 01:52:26.961066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:26.961030 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4f7628-openshift-default-dockercfg-fnt7t\"" Oct 30 01:52:27.072351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.072147 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-socket\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.072351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.072220 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-token\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.072351 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.072299 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-podinfo\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.072701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.072407 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j55fp\" (UniqueName: \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-kube-api-access-j55fp\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.072701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.072464 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-credential-socket\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.072701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.072552 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-data\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.072701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.072586 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istiod-ca-cert\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.072701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.072618 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-envoy\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.072701 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.072649 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-certs\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.179412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.178311 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-data\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.179412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.178409 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istiod-ca-cert\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.179412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.178447 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-envoy\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.179412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.178477 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-certs\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.179412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.178592 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-socket\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.179412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.178631 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-token\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.179412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.178706 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-podinfo\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.179412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.178795 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j55fp\" (UniqueName: \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-kube-api-access-j55fp\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.179412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.178849 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-credential-socket\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.201350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.182860 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-certs\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.201350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.183108 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-data\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.201350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.183734 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istiod-ca-cert\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.201350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.184774 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-socket\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.201350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.188598 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-credential-socket\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.201350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.194068 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-envoy\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.201350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.200251 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-podinfo\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.216350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.207152 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j55fp\" (UniqueName: \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-kube-api-access-j55fp\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.232915 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.232874 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-token\") pod \"e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.266097 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.265699 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:27.281317 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:27.281276 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:52:27.603936 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:52:27.603888 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd58c8f32_415f_40bd_ad3f_275b80c34f9a.slice/crio-b917a61a79de3a1515e0c53ee0fe967ce9e028c85821e9db4cf7f234a72c03c6 WatchSource:0}: Error finding container b917a61a79de3a1515e0c53ee0fe967ce9e028c85821e9db4cf7f234a72c03c6: Status 404 returned error can't find the container with id b917a61a79de3a1515e0c53ee0fe967ce9e028c85821e9db4cf7f234a72c03c6 Oct 30 01:52:27.608480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.607527 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:52:27.608480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.607624 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:52:27.608480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.607670 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:52:27.608480 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:27.608033 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd"] Oct 30 01:52:28.020260 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:28.020210 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" event={"ID":"d58c8f32-415f-40bd-ad3f-275b80c34f9a","Type":"ContainerStarted","Data":"238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d"} Oct 30 01:52:28.020726 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:28.020285 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" event={"ID":"d58c8f32-415f-40bd-ad3f-275b80c34f9a","Type":"ContainerStarted","Data":"b917a61a79de3a1515e0c53ee0fe967ce9e028c85821e9db4cf7f234a72c03c6"} Oct 30 01:52:28.064983 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:28.064925 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" podStartSLOduration=2.064908247 podStartE2EDuration="2.064908247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:28.062971401 +0000 UTC m=+2115.383928442" watchObservedRunningTime="2026-10-30 01:52:28.064908247 +0000 UTC m=+2115.385865367" Oct 30 01:52:28.267258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:28.267219 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:28.274863 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:28.274786 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:29.036416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:29.036381 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:29.038001 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:29.037975 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:52:29.463623 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:29.463585 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:52:29.956962 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:29.955545 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 01:52:29.958556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:29.957835 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 01:52:32.889790 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:32.889737 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v"] Oct 30 01:52:35.020496 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:35.020453 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:52:35.020919 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:35.020514 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:52:37.300664 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:37.300619 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:52:40.372673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.372634 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w"] Oct 30 01:52:40.377126 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.377102 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.396926 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.396892 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w"] Oct 30 01:52:40.400666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.400638 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-76a19e-serving-cert\"" Oct 30 01:52:40.522522 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.522487 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-metrics-certs\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.522778 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.522756 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.522830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.522788 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.522830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.522817 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf4zb\" (UniqueName: \"kubernetes.io/projected/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-kube-api-access-xf4zb\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.623871 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.623787 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-metrics-certs\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.624035 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.623984 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.624035 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.624008 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.624154 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.624034 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf4zb\" (UniqueName: \"kubernetes.io/projected/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-kube-api-access-xf4zb\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.624970 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.624908 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.626905 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.626882 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-maas-api-tls\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.627045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.627022 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-metrics-certs\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.646258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.646222 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf4zb\" (UniqueName: \"kubernetes.io/projected/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-kube-api-access-xf4zb\") pod \"maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:40.694563 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:40.694526 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:41.004448 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:52:41.004410 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f5d35f_45a0_4e49_aa98_65836e5d82b5.slice/crio-c2fed955c96bcc4d4edddd91e7aea262db46c2e740b4899034bbd74093cb3482 WatchSource:0}: Error finding container c2fed955c96bcc4d4edddd91e7aea262db46c2e740b4899034bbd74093cb3482: Status 404 returned error can't find the container with id c2fed955c96bcc4d4edddd91e7aea262db46c2e740b4899034bbd74093cb3482 Oct 30 01:52:41.005360 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:41.005314 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w"] Oct 30 01:52:41.006186 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:41.006160 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:52:41.138402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:41.138362 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" event={"ID":"e7f5d35f-45a0-4e49-aa98-65836e5d82b5","Type":"ContainerStarted","Data":"c2fed955c96bcc4d4edddd91e7aea262db46c2e740b4899034bbd74093cb3482"} Oct 30 01:52:42.147691 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:42.147651 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" event={"ID":"e7f5d35f-45a0-4e49-aa98-65836e5d82b5","Type":"ContainerStarted","Data":"ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2"} Oct 30 01:52:42.148230 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:42.147724 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:42.190116 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:42.190053 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" podStartSLOduration=1.857532356 podStartE2EDuration="2.190033999s" totalImagesPullingTime="332.501643ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:42.18758605 +0000 UTC m=+2129.508543090" watchObservedRunningTime="2026-10-30 01:52:42.190033999 +0000 UTC m=+2129.510991040" Oct 30 01:52:44.519760 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:44.519720 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:52:47.317192 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:47.317143 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:52:47.932832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:47.932784 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" podUID="72fa1a14-4f43-43db-99eb-55f0c55f2924" containerName="main" containerID="cri-o://d128ae2709083489239b1b375fc7f1963feb467b69e2aba8ee28b492b303df07" gracePeriod=45 Oct 30 01:52:48.164795 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.164759 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 01:52:48.194195 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.194115 5055 generic.go:356] "Generic (PLEG): container finished" podID="72fa1a14-4f43-43db-99eb-55f0c55f2924" containerID="d128ae2709083489239b1b375fc7f1963feb467b69e2aba8ee28b492b303df07" exitCode=0 Oct 30 01:52:48.194448 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.194192 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" event={"ID":"72fa1a14-4f43-43db-99eb-55f0c55f2924","Type":"ContainerDied","Data":"d128ae2709083489239b1b375fc7f1963feb467b69e2aba8ee28b492b303df07"} Oct 30 01:52:48.252481 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.252455 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:52:48.443366 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.443303 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-tmp-dir\") pod \"72fa1a14-4f43-43db-99eb-55f0c55f2924\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " Oct 30 01:52:48.443826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.443394 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-home\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-home\") pod \"72fa1a14-4f43-43db-99eb-55f0c55f2924\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " Oct 30 01:52:48.443826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.443445 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-dshm\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-dshm\") pod \"72fa1a14-4f43-43db-99eb-55f0c55f2924\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " Oct 30 01:52:48.443826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.443476 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-model-cache\") pod \"72fa1a14-4f43-43db-99eb-55f0c55f2924\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " Oct 30 01:52:48.443826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.443576 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-tmp-dir" pod "72fa1a14-4f43-43db-99eb-55f0c55f2924" (UID: "72fa1a14-4f43-43db-99eb-55f0c55f2924"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:48.443826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.443603 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72fa1a14-4f43-43db-99eb-55f0c55f2924-tls-certs\" (UniqueName: \"kubernetes.io/secret/72fa1a14-4f43-43db-99eb-55f0c55f2924-tls-certs\") pod \"72fa1a14-4f43-43db-99eb-55f0c55f2924\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " Oct 30 01:52:48.443826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.443643 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-home" pod "72fa1a14-4f43-43db-99eb-55f0c55f2924" (UID: "72fa1a14-4f43-43db-99eb-55f0c55f2924"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:48.443826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.443655 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72fa1a14-4f43-43db-99eb-55f0c55f2924-kube-api-access-96fvj\" (UniqueName: \"kubernetes.io/projected/72fa1a14-4f43-43db-99eb-55f0c55f2924-kube-api-access-96fvj\") pod \"72fa1a14-4f43-43db-99eb-55f0c55f2924\" (UID: \"72fa1a14-4f43-43db-99eb-55f0c55f2924\") " Oct 30 01:52:48.443826 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.443712 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-model-cache" pod "72fa1a14-4f43-43db-99eb-55f0c55f2924" (UID: "72fa1a14-4f43-43db-99eb-55f0c55f2924"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:48.444664 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.444605 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:48.444664 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.444643 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:48.444664 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.444658 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:48.446114 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.446089 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-dshm" pod "72fa1a14-4f43-43db-99eb-55f0c55f2924" (UID: "72fa1a14-4f43-43db-99eb-55f0c55f2924"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:48.446217 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.446113 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72fa1a14-4f43-43db-99eb-55f0c55f2924-tls-certs" pod "72fa1a14-4f43-43db-99eb-55f0c55f2924" (UID: "72fa1a14-4f43-43db-99eb-55f0c55f2924"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:52:48.446217 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.446184 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72fa1a14-4f43-43db-99eb-55f0c55f2924-kube-api-access-96fvj" pod "72fa1a14-4f43-43db-99eb-55f0c55f2924" (UID: "72fa1a14-4f43-43db-99eb-55f0c55f2924"). InnerVolumeSpecName "kube-api-access-96fvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:48.546079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.546049 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/72fa1a14-4f43-43db-99eb-55f0c55f2924-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:48.546079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.546077 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/72fa1a14-4f43-43db-99eb-55f0c55f2924-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:48.546079 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:48.546087 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96fvj\" (UniqueName: \"kubernetes.io/projected/72fa1a14-4f43-43db-99eb-55f0c55f2924-kube-api-access-96fvj\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:52:49.204185 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:49.204156 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" Oct 30 01:52:49.204434 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:49.204161 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v" event={"ID":"72fa1a14-4f43-43db-99eb-55f0c55f2924","Type":"ContainerDied","Data":"ad44bc0e42ce6daa0a8c1971d8d61049f39f1b928ffc402fbb67556a438b8fa2"} Oct 30 01:52:49.204576 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:49.204560 5055 scope.go:122] "RemoveContainer" containerID="d128ae2709083489239b1b375fc7f1963feb467b69e2aba8ee28b492b303df07" Oct 30 01:52:49.222309 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:49.222274 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:52:49.258864 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:49.258830 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v"] Oct 30 01:52:49.272626 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:49.272595 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-abb30f/auth-test-model-abb30f-kserve-7b77d5db9-tff6v"] Oct 30 01:52:50.596350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.595530 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm"] Oct 30 01:52:50.596835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.596534 5055 cpu_manager.go:414] "containerMap: removing container" podUID="72fa1a14-4f43-43db-99eb-55f0c55f2924" containerName="main" Oct 30 01:52:50.596835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.596555 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72fa1a14-4f43-43db-99eb-55f0c55f2924" containerName="main" Oct 30 01:52:50.596835 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.596773 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="72fa1a14-4f43-43db-99eb-55f0c55f2924" containerName="main" Oct 30 01:52:50.610387 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.604167 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.618361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.617376 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm"] Oct 30 01:52:50.622476 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.621399 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-92f0cc-openshift-default-dockercfg-pbhz4\"" Oct 30 01:52:50.675713 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.675677 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-certs\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.675916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.675744 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-data\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.675916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.675860 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-podinfo\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.675916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.675905 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istiod-ca-cert\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.676097 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.675986 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-envoy\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.676097 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.676065 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-token\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.676208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.676113 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4djw\" (UniqueName: \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-kube-api-access-t4djw\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.676208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.676163 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-socket\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.676311 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.676241 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-credential-socket\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.790673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.790634 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istiod-ca-cert\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.792901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.792875 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-envoy\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.793634 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.793610 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-token\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.794158 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.794137 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4djw\" (UniqueName: \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-kube-api-access-t4djw\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.794599 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.794582 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-socket\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.795097 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.795079 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-credential-socket\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.807841 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.807815 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-certs\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.808472 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.808448 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-data\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.808936 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.808917 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-podinfo\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.809175 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.794970 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-socket\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.809966 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.808315 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-certs\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.810092 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.791730 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istiod-ca-cert\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.810202 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.808833 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-data\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.810305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.801484 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-credential-socket\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.823430 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.815241 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-podinfo\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.824708 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.816544 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-envoy\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.851022 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.850948 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-token\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.851895 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.851868 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4djw\" (UniqueName: \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-kube-api-access-t4djw\") pod \"e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:50.858065 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.858030 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72fa1a14-4f43-43db-99eb-55f0c55f2924" path="/var/lib/kubelet/pods/72fa1a14-4f43-43db-99eb-55f0c55f2924/volumes" Oct 30 01:52:50.940451 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:50.940417 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:51.285910 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:52:51.285879 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda527b6b_9fe4_4c75_8fa4_e99b76442b1a.slice/crio-9f788d88cd72425f4dccc1ad74e230bcb22a0fd4156bcb248593a37e1fab8adf WatchSource:0}: Error finding container 9f788d88cd72425f4dccc1ad74e230bcb22a0fd4156bcb248593a37e1fab8adf: Status 404 returned error can't find the container with id 9f788d88cd72425f4dccc1ad74e230bcb22a0fd4156bcb248593a37e1fab8adf Oct 30 01:52:51.287130 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:51.286913 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm"] Oct 30 01:52:51.289909 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:51.289869 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:52:51.290023 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:51.289956 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:52:51.290023 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:51.289999 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:52:52.231461 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:52.231414 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" event={"ID":"da527b6b-9fe4-4c75-8fa4-e99b76442b1a","Type":"ContainerStarted","Data":"433715bc198611f37054f549f46520847e653e9fc936a6ae0b4b7163154a979b"} Oct 30 01:52:52.231923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:52.231516 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" event={"ID":"da527b6b-9fe4-4c75-8fa4-e99b76442b1a","Type":"ContainerStarted","Data":"9f788d88cd72425f4dccc1ad74e230bcb22a0fd4156bcb248593a37e1fab8adf"} Oct 30 01:52:52.297146 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:52.297089 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" podStartSLOduration=2.2970751529999998 podStartE2EDuration="2.297075153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:52.284602969 +0000 UTC m=+2139.605560010" watchObservedRunningTime="2026-10-30 01:52:52.297075153 +0000 UTC m=+2139.618032193" Oct 30 01:52:52.941283 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:52.941254 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:52.947243 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:52.947216 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:53.237912 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:53.237875 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:53.238972 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:52:53.238948 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:52:57.402285 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:57.402242 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:52:59.465358 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:52:59.465283 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:53:02.850665 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:02.850629 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-31a798"] Oct 30 01:53:02.855876 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:02.855857 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-31a798" Oct 30 01:53:02.907499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:02.907462 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-31a798"] Oct 30 01:53:02.960663 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:02.960633 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pm9kh\"" Oct 30 01:53:03.028901 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:03.028866 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlrht\" (UniqueName: \"kubernetes.io/projected/a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d-kube-api-access-xlrht\") pod \"test-curl-7498-31a798\" (UID: \"a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d\") " pod="opendatahub/test-curl-7498-31a798" Oct 30 01:53:03.130288 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:03.130196 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlrht\" (UniqueName: \"kubernetes.io/projected/a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d-kube-api-access-xlrht\") pod \"test-curl-7498-31a798\" (UID: \"a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d\") " pod="opendatahub/test-curl-7498-31a798" Oct 30 01:53:03.150985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:03.150955 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlrht\" (UniqueName: \"kubernetes.io/projected/a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d-kube-api-access-xlrht\") pod \"test-curl-7498-31a798\" (UID: \"a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d\") " pod="opendatahub/test-curl-7498-31a798" Oct 30 01:53:03.227857 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:03.227808 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-31a798" Oct 30 01:53:03.555275 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:03.555237 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-31a798"] Oct 30 01:53:03.555696 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:53:03.555665 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7e4a5a4_af3c_4a68_a4f1_8cd02e03207d.slice/crio-13844f9686fb8281c1058175c92d72f345c90b82b4f08d9886f145981a259dbb WatchSource:0}: Error finding container 13844f9686fb8281c1058175c92d72f345c90b82b4f08d9886f145981a259dbb: Status 404 returned error can't find the container with id 13844f9686fb8281c1058175c92d72f345c90b82b4f08d9886f145981a259dbb Oct 30 01:53:04.357218 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:04.357183 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-31a798" event={"ID":"a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d","Type":"ContainerStarted","Data":"a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f"} Oct 30 01:53:04.357737 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:04.357235 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-31a798" event={"ID":"a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d","Type":"ContainerStarted","Data":"13844f9686fb8281c1058175c92d72f345c90b82b4f08d9886f145981a259dbb"} Oct 30 01:53:04.410846 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:04.410791 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7498-31a798" podStartSLOduration=2.410773006 podStartE2EDuration="2.410773006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:04.409594629 +0000 UTC m=+2151.730551670" watchObservedRunningTime="2026-10-30 01:53:04.410773006 +0000 UTC m=+2151.731730053" Oct 30 01:53:05.322476 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:05.322438 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-31a798"] Oct 30 01:53:05.329714 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:05.329679 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-31a798"] Oct 30 01:53:05.576034 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:05.575942 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv"] Oct 30 01:53:05.576538 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:05.576222 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" podUID="ced403d9-79de-4102-be8a-753038340507" containerName="istio-proxy" containerID="cri-o://8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853" gracePeriod=30 Oct 30 01:53:06.373673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:06.373625 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7498-31a798" podUID="a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d" containerName="test-curl-7498-31a798" containerID="cri-o://a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f" gracePeriod=2 Oct 30 01:53:06.387442 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:53:06.387400 5055 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-31a798\" is forbidden: User \"system:node:ip-10-0-134-254.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-134-254.ec2.internal' and this object" podUID="a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d" pod="opendatahub/test-curl-7498-31a798" Oct 30 01:53:07.487746 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:53:07.487707 5055 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed9cbd9_d3ee_4100_a74b_cdbfd6165e94.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda76261f8_01b6_4764_be6b_cd36da5356ad.slice\": RecentStats: unable to find data in memory cache]" Oct 30 01:53:08.624011 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:08.623986 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-31a798" Oct 30 01:53:08.812608 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:08.812566 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d-kube-api-access-xlrht\" (UniqueName: \"kubernetes.io/projected/a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d-kube-api-access-xlrht\") pod \"a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d\" (UID: \"a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d\") " Oct 30 01:53:08.815121 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:08.815091 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d-kube-api-access-xlrht" pod "a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d" (UID: "a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d"). InnerVolumeSpecName "kube-api-access-xlrht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:08.821208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:08.821184 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d" path="/var/lib/kubelet/pods/a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d/volumes" Oct 30 01:53:08.914228 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:08.914199 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlrht\" (UniqueName: \"kubernetes.io/projected/a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d-kube-api-access-xlrht\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:09.421356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.417643 5055 generic.go:356] "Generic (PLEG): container finished" podID="a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d" containerID="a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f" exitCode=137 Oct 30 01:53:09.421356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.417930 5055 scope.go:122] "RemoveContainer" containerID="a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f" Oct 30 01:53:09.421356 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.418083 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-31a798" Oct 30 01:53:09.435548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.428625 5055 generic.go:356] "Generic (PLEG): container finished" podID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerID="d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549" exitCode=0 Oct 30 01:53:09.435548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.428686 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" event={"ID":"79831cb6-7cc2-4153-b501-079ef3bbe670","Type":"ContainerDied","Data":"d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549"} Oct 30 01:53:09.435548 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.429157 5055 scope.go:122] "RemoveContainer" containerID="d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549" Oct 30 01:53:09.458410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.443565 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:09.458410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.443669 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:09.458410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.443718 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:09.475627 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.465468 5055 scope.go:122] "RemoveContainer" containerID="a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f" Oct 30 01:53:09.504353 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:53:09.496446 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f\": container with ID starting with a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f not found: ID does not exist" containerID="a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f" Oct 30 01:53:09.504353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:09.496493 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f"} err="failed to get container status \"a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f\": rpc error: code = NotFound desc = could not find container \"a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f\": container with ID starting with a00f32f46bee9b712942bc7a27149eed7c5b13dfce744d3515ca31857934695f not found: ID does not exist" Oct 30 01:53:10.440423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:10.440386 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" event={"ID":"79831cb6-7cc2-4153-b501-079ef3bbe670","Type":"ContainerStarted","Data":"5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1"} Oct 30 01:53:10.957424 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:10.957343 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:53:11.002354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:10.998638 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:53:11.002354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:10.998779 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:53:11.002354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:10.998819 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:53:11.050989 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.050937 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-socket\") pod \"ced403d9-79de-4102-be8a-753038340507\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " Oct 30 01:53:11.051193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.051004 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ced403d9-79de-4102-be8a-753038340507-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced403d9-79de-4102-be8a-753038340507-istio-podinfo\") pod \"ced403d9-79de-4102-be8a-753038340507\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " Oct 30 01:53:11.051269 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.051212 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-credential-socket\") pod \"ced403d9-79de-4102-be8a-753038340507\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " Oct 30 01:53:11.051667 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.051637 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-credential-socket" pod "ced403d9-79de-4102-be8a-753038340507" (UID: "ced403d9-79de-4102-be8a-753038340507"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:11.052299 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.052273 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-socket" pod "ced403d9-79de-4102-be8a-753038340507" (UID: "ced403d9-79de-4102-be8a-753038340507"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:11.052469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.052447 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-certs\") pod \"ced403d9-79de-4102-be8a-753038340507\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " Oct 30 01:53:11.052705 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.052683 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-envoy\") pod \"ced403d9-79de-4102-be8a-753038340507\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " Oct 30 01:53:11.053278 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.053247 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-certs" pod "ced403d9-79de-4102-be8a-753038340507" (UID: "ced403d9-79de-4102-be8a-753038340507"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:11.053695 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.053654 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-istio-token\" (UniqueName: \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-istio-token\") pod \"ced403d9-79de-4102-be8a-753038340507\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " Oct 30 01:53:11.053808 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.053791 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-data\") pod \"ced403d9-79de-4102-be8a-753038340507\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " Oct 30 01:53:11.053876 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.053842 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced403d9-79de-4102-be8a-753038340507-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced403d9-79de-4102-be8a-753038340507-istiod-ca-cert\") pod \"ced403d9-79de-4102-be8a-753038340507\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " Oct 30 01:53:11.053936 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.053893 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-kube-api-access-q57f5\" (UniqueName: \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-kube-api-access-q57f5\") pod \"ced403d9-79de-4102-be8a-753038340507\" (UID: \"ced403d9-79de-4102-be8a-753038340507\") " Oct 30 01:53:11.054868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.054286 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-data" pod "ced403d9-79de-4102-be8a-753038340507" (UID: "ced403d9-79de-4102-be8a-753038340507"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:11.054868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.054440 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced403d9-79de-4102-be8a-753038340507-istiod-ca-cert" pod "ced403d9-79de-4102-be8a-753038340507" (UID: "ced403d9-79de-4102-be8a-753038340507"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:11.054868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.054778 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:11.054868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.054799 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:11.054868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.054813 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced403d9-79de-4102-be8a-753038340507-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:11.054868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.054825 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:11.054868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.054838 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:11.055294 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.055118 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ced403d9-79de-4102-be8a-753038340507-istio-podinfo" pod "ced403d9-79de-4102-be8a-753038340507" (UID: "ced403d9-79de-4102-be8a-753038340507"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:53:11.055819 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.055792 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-istio-token" pod "ced403d9-79de-4102-be8a-753038340507" (UID: "ced403d9-79de-4102-be8a-753038340507"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:11.057221 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.057192 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-envoy" pod "ced403d9-79de-4102-be8a-753038340507" (UID: "ced403d9-79de-4102-be8a-753038340507"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:11.057934 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.057905 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-kube-api-access-q57f5" pod "ced403d9-79de-4102-be8a-753038340507" (UID: "ced403d9-79de-4102-be8a-753038340507"). InnerVolumeSpecName "kube-api-access-q57f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:11.156619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.156581 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced403d9-79de-4102-be8a-753038340507-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:11.156619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.156611 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:11.156619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.156622 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q57f5\" (UniqueName: \"kubernetes.io/projected/ced403d9-79de-4102-be8a-753038340507-kube-api-access-q57f5\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:11.156878 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.156632 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced403d9-79de-4102-be8a-753038340507-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:11.455816 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.455777 5055 generic.go:356] "Generic (PLEG): container finished" podID="ced403d9-79de-4102-be8a-753038340507" containerID="8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853" exitCode=0 Oct 30 01:53:11.456477 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.456443 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" event={"ID":"ced403d9-79de-4102-be8a-753038340507","Type":"ContainerDied","Data":"8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853"} Oct 30 01:53:11.456619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.455868 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" Oct 30 01:53:11.456619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.456529 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv" event={"ID":"ced403d9-79de-4102-be8a-753038340507","Type":"ContainerDied","Data":"3c675628717f4cb9a52a9a6ea2a7c55b5dbba5ac441359d3a6624477a12474b5"} Oct 30 01:53:11.456619 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.456591 5055 scope.go:122] "RemoveContainer" containerID="8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853" Oct 30 01:53:11.504546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.504515 5055 scope.go:122] "RemoveContainer" containerID="8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853" Oct 30 01:53:11.505607 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:53:11.505559 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853\": container with ID starting with 8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853 not found: ID does not exist" containerID="8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853" Oct 30 01:53:11.505756 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.505617 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853"} err="failed to get container status \"8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853\": rpc error: code = NotFound desc = could not find container \"8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853\": container with ID starting with 8ffe814ca560a7dda2cc54ec3e9254c717b9b7926ccd389a56f8d3039c5b3853 not found: ID does not exist" Oct 30 01:53:11.556141 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.556109 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv"] Oct 30 01:53:11.585353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.576460 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-g9fhv"] Oct 30 01:53:11.992354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.989771 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:53:11.992354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:11.989849 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:53:12.822699 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:12.822668 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced403d9-79de-4102-be8a-753038340507" path="/var/lib/kubelet/pods/ced403d9-79de-4102-be8a-753038340507/volumes" Oct 30 01:53:12.992290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:12.991771 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:53:12.992290 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:12.991835 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:53:13.991430 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:13.989833 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:53:13.991430 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:13.989909 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:53:14.286402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.285804 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt"] Oct 30 01:53:14.309535 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.286782 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d" containerName="test-curl-7498-31a798" Oct 30 01:53:14.309535 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.286810 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d" containerName="test-curl-7498-31a798" Oct 30 01:53:14.309535 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.286868 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ced403d9-79de-4102-be8a-753038340507" containerName="istio-proxy" Oct 30 01:53:14.309535 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.286877 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced403d9-79de-4102-be8a-753038340507" containerName="istio-proxy" Oct 30 01:53:14.309535 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.287022 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced403d9-79de-4102-be8a-753038340507" containerName="istio-proxy" Oct 30 01:53:14.309535 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.287045 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e4a5a4-af3c-4a68-a4f1-8cd02e03207d" containerName="test-curl-7498-31a798" Oct 30 01:53:14.309535 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.291188 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.309535 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.308816 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2e618f-openshift-default-dockercfg-8wgkk\"" Oct 30 01:53:14.331056 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.330892 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt"] Oct 30 01:53:14.337423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.333113 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-token\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.337423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.333164 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-data\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.337423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.333198 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-envoy\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.337423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.333294 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-podinfo\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.337423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.333396 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-certs\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.337423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.333468 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-socket\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.337423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.333501 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e20bcb-70f4-416a-a159-73e28ad6e931-istiod-ca-cert\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.337423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.333539 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmnv\" (UniqueName: \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-kube-api-access-4xmnv\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.337423 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.333598 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-credential-socket\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.458888 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.458842 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-certs\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.467305 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.467214 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-certs\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.482484 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.470628 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-socket\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.483432 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.483271 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e20bcb-70f4-416a-a159-73e28ad6e931-istiod-ca-cert\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.483681 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.470938 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-socket\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.484359 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.483643 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmnv\" (UniqueName: \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-kube-api-access-4xmnv\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.490452 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.490408 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-credential-socket\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.490684 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.490657 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-credential-socket\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.698702 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.698592 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-token\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.702428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.702389 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-data\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.706637 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.706607 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-envoy\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.706963 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.706946 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-podinfo\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.711949 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.711924 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e20bcb-70f4-416a-a159-73e28ad6e931-istiod-ca-cert\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.712927 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.712906 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-data\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.713452 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.713424 5055 scope.go:122] "RemoveContainer" containerID="44bcd2874b736e0ec2ff87d79cafe68fb6b2a272afbcf2543c7a258e461e867d" Oct 30 01:53:14.718204 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.718180 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmnv\" (UniqueName: \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-kube-api-access-4xmnv\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.726382 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.726017 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-envoy\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.729798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.729718 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-podinfo\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.736818 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.736789 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-token\") pod \"e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.940485 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.940451 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:14.994378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.987497 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:53:14.994378 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:14.987555 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:53:15.371361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.369568 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt"] Oct 30 01:53:15.383362 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:53:15.379590 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e20bcb_70f4_416a_a159_73e28ad6e931.slice/crio-8d5bfeb09b4f1ae19105a3ed4d95ae6961eca294fd67ecf32442b6444da38847 WatchSource:0}: Error finding container 8d5bfeb09b4f1ae19105a3ed4d95ae6961eca294fd67ecf32442b6444da38847: Status 404 returned error can't find the container with id 8d5bfeb09b4f1ae19105a3ed4d95ae6961eca294fd67ecf32442b6444da38847 Oct 30 01:53:15.383362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.382306 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:15.383362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.382444 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:15.383362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.382491 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:15.563364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.562621 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" event={"ID":"c6e20bcb-70f4-416a-a159-73e28ad6e931","Type":"ContainerStarted","Data":"57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183"} Oct 30 01:53:15.563364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.562689 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" event={"ID":"c6e20bcb-70f4-416a-a159-73e28ad6e931","Type":"ContainerStarted","Data":"8d5bfeb09b4f1ae19105a3ed4d95ae6961eca294fd67ecf32442b6444da38847"} Oct 30 01:53:15.613412 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.612931 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" podStartSLOduration=1.61291028 podStartE2EDuration="1.61291028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:15.605854951 +0000 UTC m=+2162.926811994" watchObservedRunningTime="2026-10-30 01:53:15.61291028 +0000 UTC m=+2162.933867321" Oct 30 01:53:15.910411 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.910298 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:53:15.940832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.940785 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:15.991371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.986265 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 01:53:15.991371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:15.986350 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 01:53:16.953090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:16.953058 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:16.992624 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:16.992590 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:53:16.993928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:16.993906 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 01:53:17.578564 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:17.578509 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:17.579788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:17.579753 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:53:19.221907 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:53:19.221871 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:53:23.688037 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.687970 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4"] Oct 30 01:53:23.695030 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.694494 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.695240 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.694394 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4"] Oct 30 01:53:23.709295 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.709261 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-92f0cc-serving-cert\"" Oct 30 01:53:23.784353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.767144 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28j2l\" (UniqueName: \"kubernetes.io/projected/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-kube-api-access-28j2l\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.784353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.767208 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.784353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.767249 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.784353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.767313 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-metrics-certs\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.889353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.889119 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28j2l\" (UniqueName: \"kubernetes.io/projected/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-kube-api-access-28j2l\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.889353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.889172 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.889353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.889212 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.889353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.889266 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-metrics-certs\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.898354 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:53:23.890109 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-serving-cert: secret "maas-api-e2e-extmod-a-92f0cc-serving-cert" not found Oct 30 01:53:23.898354 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:53:23.890184 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-maas-api-tls podName:b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e nodeName:}" failed. No retries permitted until 2026-10-30 01:53:24.390163167 +0000 UTC m=+2171.711120191 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-maas-api-tls") pod "maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" (UID: "b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e") : secret "maas-api-e2e-extmod-a-92f0cc-serving-cert" not found Oct 30 01:53:23.898354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.891316 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.902713 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.902675 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-metrics-certs\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:23.912523 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:23.912494 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28j2l\" (UniqueName: \"kubernetes.io/projected/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-kube-api-access-28j2l\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:24.397350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:24.397287 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:24.400055 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:24.400027 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:24.612355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:24.611177 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:25.096619 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:53:25.096587 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c8e1a3_fb92_4dd3_83ba_a60fce315c6e.slice/crio-e53bc2577b7f056a62497c41a482f27afba06327187eb942366ba93e9ced2c43 WatchSource:0}: Error finding container e53bc2577b7f056a62497c41a482f27afba06327187eb942366ba93e9ced2c43: Status 404 returned error can't find the container with id e53bc2577b7f056a62497c41a482f27afba06327187eb942366ba93e9ced2c43 Oct 30 01:53:25.096998 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:25.096716 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4"] Oct 30 01:53:25.659285 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:25.659242 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" event={"ID":"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e","Type":"ContainerStarted","Data":"562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a"} Oct 30 01:53:25.659754 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:25.659572 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" event={"ID":"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e","Type":"ContainerStarted","Data":"e53bc2577b7f056a62497c41a482f27afba06327187eb942366ba93e9ced2c43"} Oct 30 01:53:25.659969 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:25.659953 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:25.713809 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:25.713745 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" podStartSLOduration=2.37863313 podStartE2EDuration="2.713727864s" totalImagesPullingTime="335.094734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:25.703840579 +0000 UTC m=+2173.024797620" watchObservedRunningTime="2026-10-30 01:53:25.713727864 +0000 UTC m=+2173.034684904" Oct 30 01:53:30.077546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:30.077497 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:53:30.079276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:30.077578 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:53:30.706968 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:30.706930 5055 generic.go:356] "Generic (PLEG): container finished" podID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerID="238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d" exitCode=0 Oct 30 01:53:30.707189 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:30.706988 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" event={"ID":"d58c8f32-415f-40bd-ad3f-275b80c34f9a","Type":"ContainerDied","Data":"238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d"} Oct 30 01:53:30.708135 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:30.708113 5055 scope.go:122] "RemoveContainer" containerID="238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d" Oct 30 01:53:30.712859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:30.712821 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:30.712985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:30.712921 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:30.712985 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:30.712967 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:31.674268 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:31.674237 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 01:53:31.716556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:31.716522 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" event={"ID":"d58c8f32-415f-40bd-ad3f-275b80c34f9a","Type":"ContainerStarted","Data":"775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f"} Oct 30 01:53:32.270352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:32.269761 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:53:32.270352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:32.269941 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:53:32.270352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:32.269992 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:53:33.285197 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:33.285151 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:53:33.285751 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:33.285254 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:53:33.601123 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:33.601042 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt"] Oct 30 01:53:33.601315 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:33.601272 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" containerID="cri-o://0233fb7ba8eddd6affbd30046f87eaffb0aa47ba1fcdead31443a7120a9655de" gracePeriod=30 Oct 30 01:53:34.266794 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:34.266755 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:53:34.267102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:34.267075 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:53:34.857112 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:34.857077 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5"] Oct 30 01:53:34.869645 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:34.869615 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:34.906066 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:34.906035 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d44413-openshift-default-dockercfg-7f7g9\"" Oct 30 01:53:34.935076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:34.935037 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5"] Oct 30 01:53:35.022383 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.022195 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbczd\" (UniqueName: \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-kube-api-access-sbczd\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.022383 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.022265 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46824cf2-1354-4205-8a62-21f124771343-istiod-ca-cert\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.022653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.022401 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-istio-token\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.022653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.022450 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-certs\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.022653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.022474 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-data\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.022653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.022523 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-envoy\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.022653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.022595 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-credential-socket\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.022653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.022621 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46824cf2-1354-4205-8a62-21f124771343-istio-podinfo\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.022973 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.022662 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-socket\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.124638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.123558 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-credential-socket\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.124638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.123609 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46824cf2-1354-4205-8a62-21f124771343-istio-podinfo\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.124638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.123655 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-socket\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.124638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.123775 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbczd\" (UniqueName: \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-kube-api-access-sbczd\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.124638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.123819 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46824cf2-1354-4205-8a62-21f124771343-istiod-ca-cert\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.124638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.123933 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-istio-token\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.124638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.123981 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-certs\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.124638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.124006 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-data\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.124638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.124058 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-envoy\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.135788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.132770 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-socket\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.135788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.133784 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46824cf2-1354-4205-8a62-21f124771343-istiod-ca-cert\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.135788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.134048 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-certs\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.135788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.134282 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-data\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.135788 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.134548 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-credential-socket\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.212350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.211549 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-envoy\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.260136 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.260101 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-istio-token\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.277354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.275062 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:53:35.277354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.275151 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:53:35.305354 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.304557 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46824cf2-1354-4205-8a62-21f124771343-istio-podinfo\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.317212 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.317175 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbczd\" (UniqueName: \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-kube-api-access-sbczd\") pod \"e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.580652 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.580608 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:35.949951 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.949923 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5"] Oct 30 01:53:35.951912 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:53:35.951883 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46824cf2_1354_4205_8a62_21f124771343.slice/crio-15ec7e77af5f1f9702eebf5d741c84faeeda3e24f09ff1c0c7d53fdbebcc4f6d WatchSource:0}: Error finding container 15ec7e77af5f1f9702eebf5d741c84faeeda3e24f09ff1c0c7d53fdbebcc4f6d: Status 404 returned error can't find the container with id 15ec7e77af5f1f9702eebf5d741c84faeeda3e24f09ff1c0c7d53fdbebcc4f6d Oct 30 01:53:35.954698 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.954665 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:35.954786 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.954737 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:35.954786 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:35.954774 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:36.269355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:36.266708 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:53:36.269355 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:36.266799 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:53:36.767361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:36.762242 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" event={"ID":"46824cf2-1354-4205-8a62-21f124771343","Type":"ContainerStarted","Data":"8b9ed2b5fd4ec8aa25257cad4f66ead27e9cb0c96e523a372eea5b1721c752d0"} Oct 30 01:53:36.767361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:36.762310 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" event={"ID":"46824cf2-1354-4205-8a62-21f124771343","Type":"ContainerStarted","Data":"15ec7e77af5f1f9702eebf5d741c84faeeda3e24f09ff1c0c7d53fdbebcc4f6d"} Oct 30 01:53:37.271163 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:37.271117 5055 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 30 01:53:37.271707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:37.271186 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 30 01:53:37.581638 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:37.581538 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:37.589462 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:37.589433 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:37.638598 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:37.638535 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" podStartSLOduration=3.638519116 podStartE2EDuration="3.638519116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:36.814618739 +0000 UTC m=+2184.135575779" watchObservedRunningTime="2026-10-30 01:53:37.638519116 +0000 UTC m=+2184.959476208" Oct 30 01:53:37.774629 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:37.774591 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:37.776024 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:37.775996 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:53:38.272511 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:38.272478 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:53:38.273014 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:38.272991 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:53:38.273818 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:38.273801 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 01:53:38.804777 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:38.803580 5055 generic.go:356] "Generic (PLEG): container finished" podID="24028870-9648-421c-9e8d-a196c897c985" containerID="0233fb7ba8eddd6affbd30046f87eaffb0aa47ba1fcdead31443a7120a9655de" exitCode=0 Oct 30 01:53:38.804777 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:38.804402 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" event={"ID":"24028870-9648-421c-9e8d-a196c897c985","Type":"ContainerDied","Data":"0233fb7ba8eddd6affbd30046f87eaffb0aa47ba1fcdead31443a7120a9655de"} Oct 30 01:53:38.804777 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:38.804463 5055 scope.go:122] "RemoveContainer" containerID="24852896299b26e419fbfd2f8e09c470a77ae3892d9efb6ae0afd88a703936f5" Oct 30 01:53:38.920616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:38.920590 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:53:39.030745 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.030703 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-envoy\") pod \"24028870-9648-421c-9e8d-a196c897c985\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " Oct 30 01:53:39.030959 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.030789 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24028870-9648-421c-9e8d-a196c897c985-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24028870-9648-421c-9e8d-a196c897c985-istiod-ca-cert\") pod \"24028870-9648-421c-9e8d-a196c897c985\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " Oct 30 01:53:39.030959 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.030827 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24028870-9648-421c-9e8d-a196c897c985-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24028870-9648-421c-9e8d-a196c897c985-istio-podinfo\") pod \"24028870-9648-421c-9e8d-a196c897c985\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " Oct 30 01:53:39.030959 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.030897 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-socket\") pod \"24028870-9648-421c-9e8d-a196c897c985\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " Oct 30 01:53:39.031149 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.030969 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-data\") pod \"24028870-9648-421c-9e8d-a196c897c985\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " Oct 30 01:53:39.031250 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.031226 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-socket" pod "24028870-9648-421c-9e8d-a196c897c985" (UID: "24028870-9648-421c-9e8d-a196c897c985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:39.031318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.031260 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24028870-9648-421c-9e8d-a196c897c985-istiod-ca-cert" pod "24028870-9648-421c-9e8d-a196c897c985" (UID: "24028870-9648-421c-9e8d-a196c897c985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:53:39.031602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.031391 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-kube-api-access-d47gr\" (UniqueName: \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-kube-api-access-d47gr\") pod \"24028870-9648-421c-9e8d-a196c897c985\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " Oct 30 01:53:39.031602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.031534 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-data" pod "24028870-9648-421c-9e8d-a196c897c985" (UID: "24028870-9648-421c-9e8d-a196c897c985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:39.031602 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.031536 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-istio-token\" (UniqueName: \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-istio-token\") pod \"24028870-9648-421c-9e8d-a196c897c985\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " Oct 30 01:53:39.034216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.031639 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-certs\") pod \"24028870-9648-421c-9e8d-a196c897c985\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " Oct 30 01:53:39.034216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.031716 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-credential-socket\") pod \"24028870-9648-421c-9e8d-a196c897c985\" (UID: \"24028870-9648-421c-9e8d-a196c897c985\") " Oct 30 01:53:39.034216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.031958 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-certs" pod "24028870-9648-421c-9e8d-a196c897c985" (UID: "24028870-9648-421c-9e8d-a196c897c985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:39.034216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.032155 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-credential-socket" pod "24028870-9648-421c-9e8d-a196c897c985" (UID: "24028870-9648-421c-9e8d-a196c897c985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:39.034216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.032826 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:39.034216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.032849 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:39.034216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.032864 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:39.034216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.032877 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:39.034216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.032891 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24028870-9648-421c-9e8d-a196c897c985-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:39.034760 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.034542 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24028870-9648-421c-9e8d-a196c897c985-istio-podinfo" pod "24028870-9648-421c-9e8d-a196c897c985" (UID: "24028870-9648-421c-9e8d-a196c897c985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:53:39.036746 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.036717 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-istio-token" pod "24028870-9648-421c-9e8d-a196c897c985" (UID: "24028870-9648-421c-9e8d-a196c897c985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:39.036746 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.036734 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-envoy" pod "24028870-9648-421c-9e8d-a196c897c985" (UID: "24028870-9648-421c-9e8d-a196c897c985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:39.036949 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.036930 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-kube-api-access-d47gr" pod "24028870-9648-421c-9e8d-a196c897c985" (UID: "24028870-9648-421c-9e8d-a196c897c985"). InnerVolumeSpecName "kube-api-access-d47gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:39.134727 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.134691 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24028870-9648-421c-9e8d-a196c897c985-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:39.134727 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.134722 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24028870-9648-421c-9e8d-a196c897c985-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:39.134727 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.134733 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d47gr\" (UniqueName: \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-kube-api-access-d47gr\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:39.135013 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.134743 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24028870-9648-421c-9e8d-a196c897c985-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:39.814185 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.814157 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" Oct 30 01:53:39.814729 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.814204 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt" event={"ID":"24028870-9648-421c-9e8d-a196c897c985","Type":"ContainerDied","Data":"93a337c780073192df6316cf635ef59eba8d49f18f230d81aedea4b3cb6c5493"} Oct 30 01:53:39.814729 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.814259 5055 scope.go:122] "RemoveContainer" containerID="0233fb7ba8eddd6affbd30046f87eaffb0aa47ba1fcdead31443a7120a9655de" Oct 30 01:53:39.867660 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.867630 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt"] Oct 30 01:53:39.879830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:39.879795 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8qtbt"] Oct 30 01:53:40.824502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:40.824462 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24028870-9648-421c-9e8d-a196c897c985" path="/var/lib/kubelet/pods/24028870-9648-421c-9e8d-a196c897c985/volumes" Oct 30 01:53:41.842950 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:41.842917 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8"] Oct 30 01:53:49.280519 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:53:49.280479 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:53:49.327232 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.327197 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8"] Oct 30 01:53:49.327941 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.327920 5055 cpu_manager.go:414] "containerMap: removing container" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" Oct 30 01:53:49.327941 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.327941 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" Oct 30 01:53:49.328149 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.328136 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" Oct 30 01:53:49.328209 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.328158 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" Oct 30 01:53:49.331738 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.331716 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.369661 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.369550 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8"] Oct 30 01:53:49.393156 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.393122 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-2e618f-serving-cert\"" Oct 30 01:53:49.483696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.483655 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ecaa40-4720-4849-8d03-892807164526-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.484016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.483910 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f57n\" (UniqueName: \"kubernetes.io/projected/64ecaa40-4720-4849-8d03-892807164526-kube-api-access-8f57n\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.484016 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.483986 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-metrics-certs\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.484219 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.484060 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.585223 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.585125 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f57n\" (UniqueName: \"kubernetes.io/projected/64ecaa40-4720-4849-8d03-892807164526-kube-api-access-8f57n\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.585223 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.585201 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-metrics-certs\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.585519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.585233 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.585519 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.585303 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ecaa40-4720-4849-8d03-892807164526-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.586120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.586060 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ecaa40-4720-4849-8d03-892807164526-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.588830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.588806 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-metrics-certs\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.588951 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.588855 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-maas-api-tls\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.612258 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.612023 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f57n\" (UniqueName: \"kubernetes.io/projected/64ecaa40-4720-4849-8d03-892807164526-kube-api-access-8f57n\") pod \"maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.646975 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.646940 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:49.977778 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:53:49.977745 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ecaa40_4720_4849_8d03_892807164526.slice/crio-81ee7a7eb4cf2345ebef64a769a3e5d65339c0e795cc9d09cec177afb9358c79 WatchSource:0}: Error finding container 81ee7a7eb4cf2345ebef64a769a3e5d65339c0e795cc9d09cec177afb9358c79: Status 404 returned error can't find the container with id 81ee7a7eb4cf2345ebef64a769a3e5d65339c0e795cc9d09cec177afb9358c79 Oct 30 01:53:49.978254 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:49.978229 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8"] Oct 30 01:53:50.007575 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:50.007537 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" event={"ID":"64ecaa40-4720-4849-8d03-892807164526","Type":"ContainerStarted","Data":"81ee7a7eb4cf2345ebef64a769a3e5d65339c0e795cc9d09cec177afb9358c79"} Oct 30 01:53:51.018237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:51.018198 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" event={"ID":"64ecaa40-4720-4849-8d03-892807164526","Type":"ContainerStarted","Data":"156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7"} Oct 30 01:53:51.018722 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:51.018299 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:51.062626 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:51.062561 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" podStartSLOduration=1.713543837 podStartE2EDuration="2.062540631s" totalImagesPullingTime="348.996794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:51.057372876 +0000 UTC m=+2198.378329918" watchObservedRunningTime="2026-10-30 01:53:51.062540631 +0000 UTC m=+2198.383497672" Oct 30 01:53:52.026469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:52.026424 5055 generic.go:356] "Generic (PLEG): container finished" podID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerID="433715bc198611f37054f549f46520847e653e9fc936a6ae0b4b7163154a979b" exitCode=0 Oct 30 01:53:52.027041 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:52.026506 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" event={"ID":"da527b6b-9fe4-4c75-8fa4-e99b76442b1a","Type":"ContainerDied","Data":"433715bc198611f37054f549f46520847e653e9fc936a6ae0b4b7163154a979b"} Oct 30 01:53:52.027041 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:52.027009 5055 scope.go:122] "RemoveContainer" containerID="433715bc198611f37054f549f46520847e653e9fc936a6ae0b4b7163154a979b" Oct 30 01:53:52.028866 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:52.028833 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:52.028964 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:52.028930 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:52.029006 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:52.028973 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:53:53.034761 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:53.034677 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" event={"ID":"da527b6b-9fe4-4c75-8fa4-e99b76442b1a","Type":"ContainerStarted","Data":"d614131fa1de24eaa1c0c708b4fa41b8fa4b36579f1d7d82d4ac71154f6e2bd8"} Oct 30 01:53:53.237833 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:53.237799 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:53:53.941492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:53.941461 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:53:53.942607 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:53.942572 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:53:53.942727 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:53.942630 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:53:54.941116 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:54.941075 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:53:54.941608 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:54.941146 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:53:55.941102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:55.941068 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:53:55.941288 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:55.941120 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:53:56.894798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:56.894727 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" podUID="deb6acec-32cb-4e83-b9a0-298dc297374c" containerName="main" containerID="cri-o://0b855d97340715605a5c1b38f307ad619f8bedf9753fbc4408e6e634ff00446d" gracePeriod=45 Oct 30 01:53:56.941045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:56.941012 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:53:56.941237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:56.941065 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:53:57.033024 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.032994 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 01:53:57.088350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.085121 5055 generic.go:356] "Generic (PLEG): container finished" podID="deb6acec-32cb-4e83-b9a0-298dc297374c" containerID="0b855d97340715605a5c1b38f307ad619f8bedf9753fbc4408e6e634ff00446d" exitCode=0 Oct 30 01:53:57.088350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.085257 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" event={"ID":"deb6acec-32cb-4e83-b9a0-298dc297374c","Type":"ContainerDied","Data":"0b855d97340715605a5c1b38f307ad619f8bedf9753fbc4408e6e634ff00446d"} Oct 30 01:53:57.334318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.334285 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:53:57.441253 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.441219 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-tmp-dir\") pod \"deb6acec-32cb-4e83-b9a0-298dc297374c\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " Oct 30 01:53:57.441483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.441265 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/deb6acec-32cb-4e83-b9a0-298dc297374c-tls-certs\" (UniqueName: \"kubernetes.io/secret/deb6acec-32cb-4e83-b9a0-298dc297374c-tls-certs\") pod \"deb6acec-32cb-4e83-b9a0-298dc297374c\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " Oct 30 01:53:57.441483 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.441420 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-model-cache\") pod \"deb6acec-32cb-4e83-b9a0-298dc297374c\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " Oct 30 01:53:57.441565 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.441521 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-dshm\") pod \"deb6acec-32cb-4e83-b9a0-298dc297374c\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " Oct 30 01:53:57.441609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.441569 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deb6acec-32cb-4e83-b9a0-298dc297374c-kube-api-access-9zp7v\" (UniqueName: \"kubernetes.io/projected/deb6acec-32cb-4e83-b9a0-298dc297374c-kube-api-access-9zp7v\") pod \"deb6acec-32cb-4e83-b9a0-298dc297374c\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " Oct 30 01:53:57.441653 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.441603 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-tmp-dir" pod "deb6acec-32cb-4e83-b9a0-298dc297374c" (UID: "deb6acec-32cb-4e83-b9a0-298dc297374c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:57.441730 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.441704 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-model-cache" pod "deb6acec-32cb-4e83-b9a0-298dc297374c" (UID: "deb6acec-32cb-4e83-b9a0-298dc297374c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:57.441800 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.441732 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-home\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-home\") pod \"deb6acec-32cb-4e83-b9a0-298dc297374c\" (UID: \"deb6acec-32cb-4e83-b9a0-298dc297374c\") " Oct 30 01:53:57.442026 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.441930 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-home" pod "deb6acec-32cb-4e83-b9a0-298dc297374c" (UID: "deb6acec-32cb-4e83-b9a0-298dc297374c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:57.442287 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.442239 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:57.442287 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.442261 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:57.442287 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.442269 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:57.444024 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.443996 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deb6acec-32cb-4e83-b9a0-298dc297374c-kube-api-access-9zp7v" pod "deb6acec-32cb-4e83-b9a0-298dc297374c" (UID: "deb6acec-32cb-4e83-b9a0-298dc297374c"). InnerVolumeSpecName "kube-api-access-9zp7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:57.444120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.444027 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-dshm" pod "deb6acec-32cb-4e83-b9a0-298dc297374c" (UID: "deb6acec-32cb-4e83-b9a0-298dc297374c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:57.444120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.444046 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deb6acec-32cb-4e83-b9a0-298dc297374c-tls-certs" pod "deb6acec-32cb-4e83-b9a0-298dc297374c" (UID: "deb6acec-32cb-4e83-b9a0-298dc297374c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:57.543925 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.543888 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/deb6acec-32cb-4e83-b9a0-298dc297374c-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:57.543925 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.543925 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zp7v\" (UniqueName: \"kubernetes.io/projected/deb6acec-32cb-4e83-b9a0-298dc297374c-kube-api-access-9zp7v\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:57.544172 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.543936 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/deb6acec-32cb-4e83-b9a0-298dc297374c-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:53:57.940974 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.940934 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Oct 30 01:53:57.941155 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:57.940988 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Oct 30 01:53:58.099916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:58.099884 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" Oct 30 01:53:58.099916 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:58.099905 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8" event={"ID":"deb6acec-32cb-4e83-b9a0-298dc297374c","Type":"ContainerDied","Data":"432fab01875e376e44a490a4702b68591eb0d9c0d26e2d53166e05137feaa089"} Oct 30 01:53:58.100547 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:58.099973 5055 scope.go:122] "RemoveContainer" containerID="0b855d97340715605a5c1b38f307ad619f8bedf9753fbc4408e6e634ff00446d" Oct 30 01:53:58.156768 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:58.156734 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8"] Oct 30 01:53:58.189720 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:58.189679 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-657fba/auth-test-model-657fba-kserve-5cc8d4d65c-skpj8"] Oct 30 01:53:58.821494 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:58.821456 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deb6acec-32cb-4e83-b9a0-298dc297374c" path="/var/lib/kubelet/pods/deb6acec-32cb-4e83-b9a0-298dc297374c/volumes" Oct 30 01:53:58.947133 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:58.947099 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:53:58.948147 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:53:58.948124 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 01:54:12.679759 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.679720 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p"] Oct 30 01:54:12.680494 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.680476 5055 cpu_manager.go:414] "containerMap: removing container" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" Oct 30 01:54:12.680494 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.680494 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24028870-9648-421c-9e8d-a196c897c985" containerName="istio-proxy" Oct 30 01:54:12.680622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.680516 5055 cpu_manager.go:414] "containerMap: removing container" podUID="deb6acec-32cb-4e83-b9a0-298dc297374c" containerName="main" Oct 30 01:54:12.680622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.680521 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deb6acec-32cb-4e83-b9a0-298dc297374c" containerName="main" Oct 30 01:54:12.680733 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.680637 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="deb6acec-32cb-4e83-b9a0-298dc297374c" containerName="main" Oct 30 01:54:12.684617 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.684593 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.709849 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.709808 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ww7p\" (UniqueName: \"kubernetes.io/projected/7baef220-6084-416e-a7c2-c86722049b2f-kube-api-access-6ww7p\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.710059 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.709887 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7baef220-6084-416e-a7c2-c86722049b2f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.710059 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.709928 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-metrics-certs\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.710185 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.710091 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.713174 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.713145 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-d44413-serving-cert\"" Oct 30 01:54:12.717974 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.717944 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p"] Oct 30 01:54:12.811372 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.811314 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.811562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.811462 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ww7p\" (UniqueName: \"kubernetes.io/projected/7baef220-6084-416e-a7c2-c86722049b2f-kube-api-access-6ww7p\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.811562 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.811545 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7baef220-6084-416e-a7c2-c86722049b2f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.811655 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.811596 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-metrics-certs\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.812379 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.812297 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7baef220-6084-416e-a7c2-c86722049b2f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.814185 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.814161 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-metrics-certs\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.814408 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.814384 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.831556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.831520 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ww7p\" (UniqueName: \"kubernetes.io/projected/7baef220-6084-416e-a7c2-c86722049b2f-kube-api-access-6ww7p\") pod \"maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:12.999045 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:12.999014 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:13.305474 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:54:13.305444 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7baef220_6084_416e_a7c2_c86722049b2f.slice/crio-e957905b5b1a89cb9f07b97efa7a042a8e53b6654ea133bb4c77e546792061a8 WatchSource:0}: Error finding container e957905b5b1a89cb9f07b97efa7a042a8e53b6654ea133bb4c77e546792061a8: Status 404 returned error can't find the container with id e957905b5b1a89cb9f07b97efa7a042a8e53b6654ea133bb4c77e546792061a8 Oct 30 01:54:13.307472 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:13.307441 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p"] Oct 30 01:54:13.352207 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:13.352170 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" event={"ID":"7baef220-6084-416e-a7c2-c86722049b2f","Type":"ContainerStarted","Data":"e957905b5b1a89cb9f07b97efa7a042a8e53b6654ea133bb4c77e546792061a8"} Oct 30 01:54:14.360095 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:14.360057 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" event={"ID":"7baef220-6084-416e-a7c2-c86722049b2f","Type":"ContainerStarted","Data":"ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc"} Oct 30 01:54:14.360563 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:14.360260 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:14.400124 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:14.400062 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" podStartSLOduration=1.991807873 podStartE2EDuration="2.400046913s" totalImagesPullingTime="408.23904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:14.398166414 +0000 UTC m=+2221.719123453" watchObservedRunningTime="2026-10-30 01:54:14.400046913 +0000 UTC m=+2221.721003953" Oct 30 01:54:19.226488 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:54:19.226452 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:54:20.389570 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:20.389508 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 01:54:21.007225 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.007187 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2"] Oct 30 01:54:21.014376 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.014352 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.032762 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.032001 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-0835dd-serving-cert\"" Oct 30 01:54:21.033221 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.033192 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2"] Oct 30 01:54:21.157724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.157678 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-metrics-certs\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.157911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.157752 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-maas-api-tls\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.157911 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.157896 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7bd7\" (UniqueName: \"kubernetes.io/projected/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-kube-api-access-z7bd7\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.158008 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.157938 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-openshift-service-ca\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.259654 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.259567 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7bd7\" (UniqueName: \"kubernetes.io/projected/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-kube-api-access-z7bd7\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.259654 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.259609 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-openshift-service-ca\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.259879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.259765 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-metrics-certs\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.259879 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.259793 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-maas-api-tls\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.260037 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:54:21.260011 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-serving-cert: secret "maas-api-e2e-autores-0835dd-serving-cert" not found Oct 30 01:54:21.260153 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:54:21.260124 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-maas-api-tls podName:46c3ccb9-ef66-4f59-831f-e8edce6e4b0c nodeName:}" failed. No retries permitted until 2026-10-30 01:54:21.760099763 +0000 UTC m=+2229.081056803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-maas-api-tls") pod "maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" (UID: "46c3ccb9-ef66-4f59-831f-e8edce6e4b0c") : secret "maas-api-e2e-autores-0835dd-serving-cert" not found Oct 30 01:54:21.260415 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.260391 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-openshift-service-ca\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.304492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.304446 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-metrics-certs\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.305069 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.305042 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7bd7\" (UniqueName: \"kubernetes.io/projected/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-kube-api-access-z7bd7\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.790662 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.790614 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-maas-api-tls\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.800977 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.800906 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-maas-api-tls\") pod \"maas-api-e2e-autores-0835dd-77d7f8769d-k94j2\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:21.932027 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:21.931863 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:22.306931 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:22.306894 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2"] Oct 30 01:54:22.445363 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:22.445307 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" event={"ID":"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c","Type":"ContainerStarted","Data":"abac794ec2d06c0616e8c0bb9099b16966453eaddba158c990a5a748984931fb"} Oct 30 01:54:23.453594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:23.453549 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" event={"ID":"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c","Type":"ContainerStarted","Data":"dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380"} Oct 30 01:54:23.454044 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:23.453626 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:23.506542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:23.506480 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" podStartSLOduration=3.098196166 podStartE2EDuration="3.506466438s" totalImagesPullingTime="408.270272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:23.504974051 +0000 UTC m=+2230.825931093" watchObservedRunningTime="2026-10-30 01:54:23.506466438 +0000 UTC m=+2230.827423478" Oct 30 01:54:29.465976 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:29.465943 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:54:29.504613 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:29.504580 5055 generic.go:356] "Generic (PLEG): container finished" podID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerID="57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183" exitCode=0 Oct 30 01:54:29.504818 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:29.504657 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" event={"ID":"c6e20bcb-70f4-416a-a159-73e28ad6e931","Type":"ContainerDied","Data":"57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183"} Oct 30 01:54:29.505239 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:29.505215 5055 scope.go:122] "RemoveContainer" containerID="57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183" Oct 30 01:54:29.507374 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:29.507317 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:54:29.507502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:29.507428 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:54:29.507502 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:29.507479 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:54:30.515439 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:30.515400 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" event={"ID":"c6e20bcb-70f4-416a-a159-73e28ad6e931","Type":"ContainerStarted","Data":"957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944"} Oct 30 01:54:30.940928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:30.940839 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:54:30.942446 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:30.942416 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:54:30.942594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:30.942468 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:54:31.941166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:31.941133 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:54:31.941632 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:31.941197 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:54:32.582353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:32.578440 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:54:32.954511 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:32.953504 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:54:32.954511 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:32.953562 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:54:33.942019 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:33.941976 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:54:33.942246 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:33.942044 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:54:34.941259 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:34.941225 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Oct 30 01:54:34.941707 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:34.941295 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Oct 30 01:54:36.156489 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:36.156458 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:54:36.158875 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:36.158851 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 01:54:43.001157 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.001117 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq"] Oct 30 01:54:43.006616 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.006591 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.027571 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.027534 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-f1edaf-serving-cert\"" Oct 30 01:54:43.030814 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.030784 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq"] Oct 30 01:54:43.097673 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.097638 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.097871 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.097809 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-metrics-certs\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.097939 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.097873 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmphj\" (UniqueName: \"kubernetes.io/projected/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-kube-api-access-tmphj\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.098043 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.098025 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.205282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.204414 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.205282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.204520 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-metrics-certs\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.205282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.204551 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmphj\" (UniqueName: \"kubernetes.io/projected/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-kube-api-access-tmphj\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.205282 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.204657 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.205839 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:54:43.205572 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-serving-cert: secret "maas-api-e2e-extmod-b-f1edaf-serving-cert" not found Oct 30 01:54:43.205839 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:54:43.205680 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-maas-api-tls podName:3c52f82e-cc60-4e4a-84a5-0f224b1c63f0 nodeName:}" failed. No retries permitted until 2026-10-30 01:54:43.705658787 +0000 UTC m=+2251.026615821 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-maas-api-tls") pod "maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" (UID: "3c52f82e-cc60-4e4a-84a5-0f224b1c63f0") : secret "maas-api-e2e-extmod-b-f1edaf-serving-cert" not found Oct 30 01:54:43.207087 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.207040 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.218674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.218643 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-metrics-certs\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.229774 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.229736 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmphj\" (UniqueName: \"kubernetes.io/projected/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-kube-api-access-tmphj\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.714609 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.714572 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.717232 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.717210 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:43.922618 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:43.922570 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:44.274157 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:54:44.274127 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c52f82e_cc60_4e4a_84a5_0f224b1c63f0.slice/crio-f03b3661205e1c58b23a31e749e937fa333bf6895ad499c2745a5f36958cae40 WatchSource:0}: Error finding container f03b3661205e1c58b23a31e749e937fa333bf6895ad499c2745a5f36958cae40: Status 404 returned error can't find the container with id f03b3661205e1c58b23a31e749e937fa333bf6895ad499c2745a5f36958cae40 Oct 30 01:54:44.277160 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:44.277137 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq"] Oct 30 01:54:44.649689 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:44.649609 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" event={"ID":"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0","Type":"ContainerStarted","Data":"f03b3661205e1c58b23a31e749e937fa333bf6895ad499c2745a5f36958cae40"} Oct 30 01:54:45.657582 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:45.657535 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" event={"ID":"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0","Type":"ContainerStarted","Data":"b6ae10684a05f1443368c2693fe43d1400b805e4717934b33cdf3614c3de0b4d"} Oct 30 01:54:45.658032 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:45.657640 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:45.697781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:45.697711 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" podStartSLOduration=3.285859899 podStartE2EDuration="3.697696294s" totalImagesPullingTime="411.836395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:45.696394921 +0000 UTC m=+2253.017351961" watchObservedRunningTime="2026-10-30 01:54:45.697696294 +0000 UTC m=+2253.018653334" Oct 30 01:54:49.242374 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:54:49.242313 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:54:51.688748 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:51.688713 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 01:54:57.760510 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:57.760472 5055 generic.go:356] "Generic (PLEG): container finished" podID="46824cf2-1354-4205-8a62-21f124771343" containerID="8b9ed2b5fd4ec8aa25257cad4f66ead27e9cb0c96e523a372eea5b1721c752d0" exitCode=0 Oct 30 01:54:57.761304 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:57.761275 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" event={"ID":"46824cf2-1354-4205-8a62-21f124771343","Type":"ContainerDied","Data":"8b9ed2b5fd4ec8aa25257cad4f66ead27e9cb0c96e523a372eea5b1721c752d0"} Oct 30 01:54:57.761974 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:57.761947 5055 scope.go:122] "RemoveContainer" containerID="8b9ed2b5fd4ec8aa25257cad4f66ead27e9cb0c96e523a372eea5b1721c752d0" Oct 30 01:54:57.764495 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:57.764455 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:54:57.764930 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:57.764904 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:54:57.765027 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:57.764982 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:54:58.772975 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:58.772933 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" event={"ID":"46824cf2-1354-4205-8a62-21f124771343","Type":"ContainerStarted","Data":"66cc112c59ed1e4d6659c8499f9f41b286f726be3aff3da77d0193bef280a496"} Oct 30 01:54:59.584350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:59.583736 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:54:59.584350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:59.583872 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:54:59.584350 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:54:59.583930 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:55:00.586000 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:00.585957 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:55:00.586696 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:00.586663 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:55:01.593353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:01.592743 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:55:01.593353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:01.592807 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:55:02.582353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:02.581911 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:55:02.582353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:02.581974 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:55:03.585468 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:03.585434 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:55:03.586091 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:03.586060 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:55:04.585811 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:04.585719 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Oct 30 01:55:04.586428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:04.586382 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Oct 30 01:55:05.587216 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:05.587186 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:55:05.587954 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:05.587798 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:55:05.589184 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:05.589164 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 01:55:06.548015 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.547982 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz"] Oct 30 01:55:06.554761 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.554736 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.598682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.590831 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz"] Oct 30 01:55:06.603397 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.603181 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-cd8173-openshift-default-dockercfg-hw9jg\"" Oct 30 01:55:06.607479 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.607457 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvpd\" (UniqueName: \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-kube-api-access-sgvpd\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.607670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.607654 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc69473a-f21a-4e8a-83da-2f2469a423bd-istiod-ca-cert\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.607801 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.607789 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-socket\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.607928 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.607916 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-credential-socket\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.608031 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.608020 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-envoy\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.608166 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.608151 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-token\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.608317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.608303 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-certs\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.608452 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.608440 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-podinfo\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.608593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.608581 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-data\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.710283 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.710244 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-token\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.710487 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.710318 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-certs\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.710487 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.710360 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-podinfo\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.710487 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.710413 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-data\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.710680 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.710562 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvpd\" (UniqueName: \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-kube-api-access-sgvpd\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.710680 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.710618 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc69473a-f21a-4e8a-83da-2f2469a423bd-istiod-ca-cert\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.710680 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.710659 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-socket\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.710853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.710709 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-credential-socket\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.710853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.710734 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-envoy\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.710853 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.710826 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-certs\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.711131 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.711107 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-data\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.711542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.711479 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-credential-socket\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.711542 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.711501 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-socket\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.711774 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.711745 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc69473a-f21a-4e8a-83da-2f2469a423bd-istiod-ca-cert\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.713896 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.713872 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-envoy\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.718851 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.718821 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-podinfo\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.730779 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.730745 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-token\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.731132 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.731113 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvpd\" (UniqueName: \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-kube-api-access-sgvpd\") pod \"e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:06.917546 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:06.914009 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:07.347167 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:07.346874 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz"] Oct 30 01:55:07.351738 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:07.351703 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:55:07.351864 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:07.351768 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:55:07.351864 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:07.351795 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:55:07.889126 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:07.889083 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" event={"ID":"fc69473a-f21a-4e8a-83da-2f2469a423bd","Type":"ContainerStarted","Data":"bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85"} Oct 30 01:55:07.889663 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:07.889151 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" event={"ID":"fc69473a-f21a-4e8a-83da-2f2469a423bd","Type":"ContainerStarted","Data":"eb23b5b9d9667c3f77e1e8e7cd215c14e09b09592e9b53f03634484064af2c01"} Oct 30 01:55:07.914762 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:07.914719 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:07.962481 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:07.962423 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" podStartSLOduration=1.962403123 podStartE2EDuration="1.962403123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:07.945500238 +0000 UTC m=+2275.266457276" watchObservedRunningTime="2026-10-30 01:55:07.962403123 +0000 UTC m=+2275.283360165" Oct 30 01:55:08.921460 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:08.921429 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:09.904868 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:09.904838 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:09.905762 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:55:09.905741 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:55:19.220682 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:55:19.220645 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:55:49.221384 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:55:49.221281 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:56:17.989882 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:17.989837 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2"] Oct 30 01:56:17.990833 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:17.990600 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" podUID="46c3ccb9-ef66-4f59-831f-e8edce6e4b0c" containerName="maas-api" containerID="cri-o://dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380" gracePeriod=30 Oct 30 01:56:18.541438 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.541408 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:56:18.541650 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.541505 5055 generic.go:356] "Generic (PLEG): container finished" podID="46c3ccb9-ef66-4f59-831f-e8edce6e4b0c" containerID="dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380" exitCode=0 Oct 30 01:56:18.541650 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.541563 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" event={"ID":"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c","Type":"ContainerDied","Data":"dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380"} Oct 30 01:56:18.541650 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.541614 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" event={"ID":"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c","Type":"ContainerDied","Data":"abac794ec2d06c0616e8c0bb9099b16966453eaddba158c990a5a748984931fb"} Oct 30 01:56:18.541650 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.541645 5055 scope.go:122] "RemoveContainer" containerID="dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380" Oct 30 01:56:18.554027 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.554007 5055 scope.go:122] "RemoveContainer" containerID="dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380" Oct 30 01:56:18.554370 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:56:18.554349 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380\": container with ID starting with dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380 not found: ID does not exist" containerID="dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380" Oct 30 01:56:18.554441 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.554379 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380"} err="failed to get container status \"dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380\": rpc error: code = NotFound desc = could not find container \"dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380\": container with ID starting with dcd2d8e47b56c89343085a62cd98830989aa9a9ad73235e6ff4c41a2935e9380 not found: ID does not exist" Oct 30 01:56:18.610572 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.610505 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-openshift-service-ca\") pod \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " Oct 30 01:56:18.610768 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.610738 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-kube-api-access-z7bd7\" (UniqueName: \"kubernetes.io/projected/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-kube-api-access-z7bd7\") pod \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " Oct 30 01:56:18.610837 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.610812 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-maas-api-tls\") pod \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " Oct 30 01:56:18.610945 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.610927 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-metrics-certs\") pod \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\" (UID: \"46c3ccb9-ef66-4f59-831f-e8edce6e4b0c\") " Oct 30 01:56:18.611007 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.610953 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-openshift-service-ca" pod "46c3ccb9-ef66-4f59-831f-e8edce6e4b0c" (UID: "46c3ccb9-ef66-4f59-831f-e8edce6e4b0c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:56:18.611775 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.611729 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:56:18.614314 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.614289 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-maas-api-tls" pod "46c3ccb9-ef66-4f59-831f-e8edce6e4b0c" (UID: "46c3ccb9-ef66-4f59-831f-e8edce6e4b0c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:56:18.614492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.614387 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-metrics-certs" pod "46c3ccb9-ef66-4f59-831f-e8edce6e4b0c" (UID: "46c3ccb9-ef66-4f59-831f-e8edce6e4b0c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:56:18.615594 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.615568 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-kube-api-access-z7bd7" pod "46c3ccb9-ef66-4f59-831f-e8edce6e4b0c" (UID: "46c3ccb9-ef66-4f59-831f-e8edce6e4b0c"). InnerVolumeSpecName "kube-api-access-z7bd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:56:18.713060 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.713028 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:56:18.713060 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.713061 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7bd7\" (UniqueName: \"kubernetes.io/projected/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-kube-api-access-z7bd7\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:56:18.713276 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:18.713075 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 01:56:19.221925 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:56:19.221884 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:56:19.549469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:19.549376 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2" Oct 30 01:56:19.606623 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:19.606586 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2"] Oct 30 01:56:19.618072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:19.618043 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-0835dd-77d7f8769d-k94j2"] Oct 30 01:56:20.823662 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:20.823629 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c3ccb9-ef66-4f59-831f-e8edce6e4b0c" path="/var/lib/kubelet/pods/46c3ccb9-ef66-4f59-831f-e8edce6e4b0c/volumes" Oct 30 01:56:28.607391 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.607360 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km"] Oct 30 01:56:28.609109 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.609086 5055 cpu_manager.go:414] "containerMap: removing container" podUID="46c3ccb9-ef66-4f59-831f-e8edce6e4b0c" containerName="maas-api" Oct 30 01:56:28.609256 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.609244 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c3ccb9-ef66-4f59-831f-e8edce6e4b0c" containerName="maas-api" Oct 30 01:56:28.609545 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.609533 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c3ccb9-ef66-4f59-831f-e8edce6e4b0c" containerName="maas-api" Oct 30 01:56:28.615556 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.614779 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.634592 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.634564 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-817fd7-openshift-default-dockercfg-hjjqm\"" Oct 30 01:56:28.648614 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.648578 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km"] Oct 30 01:56:28.746205 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.746164 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-token\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.746402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.746290 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istiod-ca-cert\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.746402 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.746366 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-envoy\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.746487 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.746417 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzj6s\" (UniqueName: \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-kube-api-access-tzj6s\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.746487 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.746434 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-certs\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.746615 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.746593 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-data\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.746724 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.746708 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-podinfo\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.746777 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.746761 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-credential-socket\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.746897 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.746881 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-socket\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.850895 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.850436 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-socket\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.851215 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.850869 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-socket\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.851442 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.851424 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-token\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.851915 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.851893 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istiod-ca-cert\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.852113 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.852098 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-envoy\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.871193 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.871117 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzj6s\" (UniqueName: \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-kube-api-access-tzj6s\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.871763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.871739 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-certs\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.872364 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.872345 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-data\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.873549 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.873529 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-podinfo\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.874433 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.874412 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-credential-socket\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.874670 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.873379 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-data\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.901400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.852674 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istiod-ca-cert\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.901400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.872185 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-certs\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.901400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.870760 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-envoy\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.901400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.874762 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-credential-socket\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.930353 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.928597 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-podinfo\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.931881 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.931852 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-token\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:28.954114 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:28.954075 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzj6s\" (UniqueName: \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-kube-api-access-tzj6s\") pod \"e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:29.246897 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:29.246860 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:29.608407 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:56:29.608305 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7458b5bd_13a6_4044_a5cb_66160c5dd31a.slice/crio-c43b8348886d792928f1384bffb5077745fff17a205a525401e9250cf30bb282 WatchSource:0}: Error finding container c43b8348886d792928f1384bffb5077745fff17a205a525401e9250cf30bb282: Status 404 returned error can't find the container with id c43b8348886d792928f1384bffb5077745fff17a205a525401e9250cf30bb282 Oct 30 01:56:29.609781 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:29.609756 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km"] Oct 30 01:56:29.611815 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:29.611780 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:56:29.611898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:29.611844 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:56:29.611898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:29.611875 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:56:29.636859 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:29.636831 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" event={"ID":"7458b5bd-13a6-4044-a5cb-66160c5dd31a","Type":"ContainerStarted","Data":"c43b8348886d792928f1384bffb5077745fff17a205a525401e9250cf30bb282"} Oct 30 01:56:30.644907 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:30.644869 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" event={"ID":"7458b5bd-13a6-4044-a5cb-66160c5dd31a","Type":"ContainerStarted","Data":"2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d"} Oct 30 01:56:30.702597 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:30.702535 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" podStartSLOduration=2.702515871 podStartE2EDuration="2.702515871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:30.700205794 +0000 UTC m=+2358.021162832" watchObservedRunningTime="2026-10-30 01:56:30.702515871 +0000 UTC m=+2358.023472912" Oct 30 01:56:31.247313 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:31.247271 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:31.254155 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:31.254120 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:31.652923 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:31.652839 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:31.653938 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:31.653922 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:56:40.946500 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:40.946452 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:56:40.946934 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:40.946538 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:56:43.777613 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:43.777567 5055 generic.go:356] "Generic (PLEG): container finished" podID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerID="bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85" exitCode=0 Oct 30 01:56:43.778219 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:43.777792 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" event={"ID":"fc69473a-f21a-4e8a-83da-2f2469a423bd","Type":"ContainerDied","Data":"bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85"} Oct 30 01:56:43.778410 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:43.778373 5055 scope.go:122] "RemoveContainer" containerID="bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85" Oct 30 01:56:43.780385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:43.780351 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:56:43.780469 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:43.780432 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:56:43.780507 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:43.780467 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:56:44.824385 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:44.824348 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" event={"ID":"fc69473a-f21a-4e8a-83da-2f2469a423bd","Type":"ContainerStarted","Data":"d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932"} Oct 30 01:56:44.915115 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:44.915073 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:56:44.915317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:44.915218 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:56:44.915317 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:44.915272 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:56:45.915076 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:45.915038 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:56:45.915513 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:45.915100 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:56:46.915727 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:46.915693 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:56:46.916143 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:46.915768 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:56:47.918436 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:47.916792 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:56:47.918436 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:47.916852 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:56:48.915572 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:48.915541 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:56:48.915756 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:48.915603 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:56:49.220253 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:56:49.220208 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:56:49.915100 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:49.915062 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Oct 30 01:56:49.915367 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:49.915129 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Oct 30 01:56:50.920694 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:50.920664 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:56:50.921417 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:50.921383 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:56:50.921979 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:56:50.921961 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 01:57:13.295867 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:13.295835 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:57:13.302545 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:13.302518 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 01:57:19.221225 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:57:19.221184 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:57:37.212171 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:37.212135 5055 generic.go:356] "Generic (PLEG): container finished" podID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerID="2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d" exitCode=0 Oct 30 01:57:37.212666 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:37.212209 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" event={"ID":"7458b5bd-13a6-4044-a5cb-66160c5dd31a","Type":"ContainerDied","Data":"2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d"} Oct 30 01:57:37.212734 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:37.212720 5055 scope.go:122] "RemoveContainer" containerID="2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d" Oct 30 01:57:37.214733 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:37.214705 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:57:37.214827 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:37.214776 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:57:37.214827 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:37.214818 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:57:38.222346 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:38.222295 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" event={"ID":"7458b5bd-13a6-4044-a5cb-66160c5dd31a","Type":"ContainerStarted","Data":"9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96"} Oct 30 01:57:38.247467 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:38.247440 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:57:38.247747 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:38.247723 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:57:38.247805 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:38.247773 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:57:39.248107 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:39.248076 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:57:39.248622 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:39.248128 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:57:40.247400 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:40.247372 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:57:40.247593 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:40.247429 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:57:41.248044 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:41.248010 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:57:41.248498 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:41.248079 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:57:42.247573 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:42.247537 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:57:42.247763 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:42.247598 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:57:43.247968 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:43.247935 5055 patch_prober.go:29] interesting pod/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" start-of-body= Oct 30 01:57:43.248415 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:43.248001 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.172:15021/healthz/ready\": dial tcp 10.132.0.172:15021: connect: connection refused" Oct 30 01:57:44.252715 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:44.252683 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:57:44.253120 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:44.253025 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:57:44.253713 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:57:44.253695 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 01:57:49.220386 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:57:49.220309 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:58:19.226805 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:58:19.226766 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:58:41.956944 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:41.956850 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt"] Oct 30 01:58:41.961632 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:41.961597 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:41.979284 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:41.979258 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-f6e11e-serving-cert\"" Oct 30 01:58:42.001349 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:41.997667 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt"] Oct 30 01:58:42.025510 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.025472 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-metrics-certs\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.025808 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.025781 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c10d615-5cde-47ae-821b-7e37711e71cd-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.025940 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.025852 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.026009 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.025946 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26rws\" (UniqueName: \"kubernetes.io/projected/7c10d615-5cde-47ae-821b-7e37711e71cd-kube-api-access-26rws\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.128046 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.128007 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-metrics-certs\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.128237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.128069 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c10d615-5cde-47ae-821b-7e37711e71cd-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.128237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.128106 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.128237 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.128126 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26rws\" (UniqueName: \"kubernetes.io/projected/7c10d615-5cde-47ae-821b-7e37711e71cd-kube-api-access-26rws\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.128836 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.128817 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c10d615-5cde-47ae-821b-7e37711e71cd-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.130861 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.130832 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-maas-api-tls\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.130987 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.130931 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-metrics-certs\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.148042 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.148014 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26rws\" (UniqueName: \"kubernetes.io/projected/7c10d615-5cde-47ae-821b-7e37711e71cd-kube-api-access-26rws\") pod \"maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.288163 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.288057 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:42.600012 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:58:42.599977 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c10d615_5cde_47ae_821b_7e37711e71cd.slice/crio-bc7ba623b275234bbe23f867eee581592feba7aa697090b0a0c1206d8d61967c WatchSource:0}: Error finding container bc7ba623b275234bbe23f867eee581592feba7aa697090b0a0c1206d8d61967c: Status 404 returned error can't find the container with id bc7ba623b275234bbe23f867eee581592feba7aa697090b0a0c1206d8d61967c Oct 30 01:58:42.600669 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.600555 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt"] Oct 30 01:58:42.601733 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.601713 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:58:42.898102 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:42.898009 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" event={"ID":"7c10d615-5cde-47ae-821b-7e37711e71cd","Type":"ContainerStarted","Data":"bc7ba623b275234bbe23f867eee581592feba7aa697090b0a0c1206d8d61967c"} Oct 30 01:58:43.908995 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:43.908951 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" event={"ID":"7c10d615-5cde-47ae-821b-7e37711e71cd","Type":"ContainerStarted","Data":"b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61"} Oct 30 01:58:43.909492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:43.909053 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:58:43.959177 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:43.959123 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" podStartSLOduration=2.641992409 podStartE2EDuration="2.959106971s" totalImagesPullingTime="317.114562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:43.958072625 +0000 UTC m=+2491.279029658" watchObservedRunningTime="2026-10-30 01:58:43.959106971 +0000 UTC m=+2491.280064011" Oct 30 01:58:49.223456 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:58:49.223420 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:58:49.921908 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:58:49.921872 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 01:59:19.227374 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:59:19.225259 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:59:34.691641 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.691594 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv"] Oct 30 01:59:34.696208 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.696186 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.720044 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.720012 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv"] Oct 30 01:59:34.873515 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.873463 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.873749 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.873524 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.873749 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.873652 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.873832 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.873782 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.873872 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.873855 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.873925 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.873910 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c4bj\" (UniqueName: \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-kube-api-access-6c4bj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.873988 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.873977 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.874028 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.874019 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.874065 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.874038 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.975114 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975069 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.975338 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975165 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.975338 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975197 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.975338 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975255 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.975338 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975298 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.975499 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975404 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.976371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975557 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.976371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975639 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.976371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975713 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.976371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975849 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.976371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975939 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.976371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.975983 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c4bj\" (UniqueName: \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-kube-api-access-6c4bj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.976371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.976035 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.976371 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.976229 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.978818 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.978791 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.979340 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.979290 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.996463 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.996428 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:34.997422 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:34.997390 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c4bj\" (UniqueName: \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-kube-api-access-6c4bj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-jhpgv\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:35.023152 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:35.023119 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:35.363893 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:35.363858 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv"] Oct 30 01:59:35.366014 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:59:35.365986 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e7b35c_1ce6_4b22_9c80_f6745b3727b3.slice/crio-f082b27fe8293c6ff7abc5168c3dd5f595e49b1517664444d196b318f3d48174 WatchSource:0}: Error finding container f082b27fe8293c6ff7abc5168c3dd5f595e49b1517664444d196b318f3d48174: Status 404 returned error can't find the container with id f082b27fe8293c6ff7abc5168c3dd5f595e49b1517664444d196b318f3d48174 Oct 30 01:59:35.369369 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:35.369301 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:59:35.369492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:35.369418 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:59:35.369492 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:35.369468 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:59:35.415956 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:35.415817 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" event={"ID":"34e7b35c-1ce6-4b22-9c80-f6745b3727b3","Type":"ContainerStarted","Data":"f082b27fe8293c6ff7abc5168c3dd5f595e49b1517664444d196b318f3d48174"} Oct 30 01:59:36.424458 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:36.424418 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" event={"ID":"34e7b35c-1ce6-4b22-9c80-f6745b3727b3","Type":"ContainerStarted","Data":"2c2c0a94b39cca4e37a4a11498a1aed0b502c1b7f503a2e2450084fc15a16a6c"} Oct 30 01:59:36.489352 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:36.488821 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" podStartSLOduration=2.488797071 podStartE2EDuration="2.488797071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:36.474824758 +0000 UTC m=+2543.795781799" watchObservedRunningTime="2026-10-30 01:59:36.488797071 +0000 UTC m=+2543.809754115" Oct 30 01:59:37.024019 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:37.023982 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:37.025798 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:37.025769 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jhpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:59:37.025946 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:37.025831 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:59:38.024318 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:38.024284 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jhpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:59:38.024757 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:38.024368 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:59:39.024361 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:39.024294 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jhpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:59:39.024806 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:39.024368 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:59:40.024280 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:40.024234 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jhpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:59:40.024525 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:40.024309 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:59:41.023898 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:41.023859 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jhpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:59:41.024236 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:41.023919 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:59:42.023584 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:42.023545 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jhpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:59:42.024004 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:42.023609 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:59:43.023967 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:43.023923 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-jhpgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 30 01:59:43.024416 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:43.023993 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 30 01:59:44.067949 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:44.067919 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:44.069231 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:44.069206 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:44.070285 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:44.070266 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 01:59:49.380674 ip-10-0-134-254 kubenswrapper[5055]: E1030 01:59:49.380597 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 01:59:49.737937 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.737886 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw"] Oct 30 01:59:49.752388 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.751562 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.784055 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.784021 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw"] Oct 30 01:59:49.859889 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.859836 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.860085 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.859959 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58c0b93-1a99-44ad-847d-d5022d773a51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.860085 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.860018 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zf6t\" (UniqueName: \"kubernetes.io/projected/f58c0b93-1a99-44ad-847d-d5022d773a51-kube-api-access-8zf6t\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.860085 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.860071 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.860227 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.860105 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.860227 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.860129 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.860227 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.860222 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.860362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.860247 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.860362 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.860272 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.961205 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961173 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.961428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961250 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58c0b93-1a99-44ad-847d-d5022d773a51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.961428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961278 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zf6t\" (UniqueName: \"kubernetes.io/projected/f58c0b93-1a99-44ad-847d-d5022d773a51-kube-api-access-8zf6t\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.961428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961353 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.961428 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961398 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.961674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961429 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.961674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961480 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.961674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961521 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.961674 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961560 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.962343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961982 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.962343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.961947 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.962343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.962097 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f58c0b93-1a99-44ad-847d-d5022d773a51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.962343 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.962241 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.964860 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.964832 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.965061 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.965039 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.966251 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.966233 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.983725 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.983694 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zf6t\" (UniqueName: \"kubernetes.io/projected/f58c0b93-1a99-44ad-847d-d5022d773a51-kube-api-access-8zf6t\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:49.984090 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:49.984067 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f58c0b93-1a99-44ad-847d-d5022d773a51-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-z65cw\" (UID: \"f58c0b93-1a99-44ad-847d-d5022d773a51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:50.089383 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:50.089286 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:50.435745 ip-10-0-134-254 kubenswrapper[5055]: W1030 01:59:50.435716 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58c0b93_1a99_44ad_847d_d5022d773a51.slice/crio-fce3d7081d0fb357baa3d776e2f4e708a34bd52619700e8023a97012a082e746 WatchSource:0}: Error finding container fce3d7081d0fb357baa3d776e2f4e708a34bd52619700e8023a97012a082e746: Status 404 returned error can't find the container with id fce3d7081d0fb357baa3d776e2f4e708a34bd52619700e8023a97012a082e746 Oct 30 01:59:50.438040 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:50.438016 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw"] Oct 30 01:59:50.438682 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:50.438652 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:59:50.438766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:50.438715 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:59:50.438766 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:50.438745 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 01:59:50.701684 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:50.701581 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" event={"ID":"f58c0b93-1a99-44ad-847d-d5022d773a51","Type":"ContainerStarted","Data":"6bed1377988ca80030b106fa824a4518c7baaf7c3cfdc66f3f5a51665b24246e"} Oct 30 01:59:50.701684 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:50.701652 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" event={"ID":"f58c0b93-1a99-44ad-847d-d5022d773a51","Type":"ContainerStarted","Data":"fce3d7081d0fb357baa3d776e2f4e708a34bd52619700e8023a97012a082e746"} Oct 30 01:59:50.746472 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:50.746420 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" podStartSLOduration=1.746406575 podStartE2EDuration="1.746406575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:50.742972446 +0000 UTC m=+2558.063929488" watchObservedRunningTime="2026-10-30 01:59:50.746406575 +0000 UTC m=+2558.067363615" Oct 30 01:59:51.090072 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:51.090034 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:51.091780 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:51.091746 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-z65cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:59:51.091910 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:51.091808 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" podUID="f58c0b93-1a99-44ad-847d-d5022d773a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:59:52.092307 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:52.092274 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-z65cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:59:52.092915 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:52.092885 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" podUID="f58c0b93-1a99-44ad-847d-d5022d773a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:59:53.090613 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:53.090190 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-z65cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:59:53.090613 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:53.090269 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" podUID="f58c0b93-1a99-44ad-847d-d5022d773a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:59:54.093471 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:54.093423 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-z65cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:59:54.094011 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:54.093493 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" podUID="f58c0b93-1a99-44ad-847d-d5022d773a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:59:55.090394 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:55.090359 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-z65cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:59:55.090639 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:55.090413 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" podUID="f58c0b93-1a99-44ad-847d-d5022d773a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:59:56.090729 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:56.090692 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-z65cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:59:56.091315 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:56.091287 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" podUID="f58c0b93-1a99-44ad-847d-d5022d773a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:59:57.090543 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:57.090508 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-z65cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:59:57.090725 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:57.090561 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" podUID="f58c0b93-1a99-44ad-847d-d5022d773a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:59:58.089785 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:58.089743 5055 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-z65cw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 30 01:59:58.090211 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:58.089814 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" podUID="f58c0b93-1a99-44ad-847d-d5022d773a51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 30 01:59:59.361830 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:59.361794 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:59.362273 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:59.362223 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 01:59:59.363234 ip-10-0-134-254 kubenswrapper[5055]: I1030 01:59:59.363211 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-z65cw" Oct 30 02:00:00.241604 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.241570 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d"] Oct 30 02:00:00.247527 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.247504 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" Oct 30 02:00:00.262983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.262950 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g"] Oct 30 02:00:00.274241 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.274202 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d"] Oct 30 02:00:00.274434 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.274376 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" Oct 30 02:00:00.304626 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.304596 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl"] Oct 30 02:00:00.314180 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.314144 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" Oct 30 02:00:00.319422 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.319397 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g"] Oct 30 02:00:00.323818 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.323795 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl"] Oct 30 02:00:00.357280 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.357245 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:00:00.391878 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.391746 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b114e1-248a-4a90-86a1-2490276b845d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d\" (UID: \"d2b114e1-248a-4a90-86a1-2490276b845d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" Oct 30 02:00:00.393037 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.393007 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d038f3e-e623-4011-ab10-ab444381acca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g\" (UID: \"6d038f3e-e623-4011-ab10-ab444381acca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" Oct 30 02:00:00.393161 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.393069 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq4lf\" (UniqueName: \"kubernetes.io/projected/f9ef245f-d702-4a20-95b1-281f2c80f3d3-kube-api-access-wq4lf\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl\" (UID: \"f9ef245f-d702-4a20-95b1-281f2c80f3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" Oct 30 02:00:00.393240 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.393173 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ef245f-d702-4a20-95b1-281f2c80f3d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl\" (UID: \"f9ef245f-d702-4a20-95b1-281f2c80f3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" Oct 30 02:00:00.393296 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.393277 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vwgj\" (UniqueName: \"kubernetes.io/projected/d2b114e1-248a-4a90-86a1-2490276b845d-kube-api-access-4vwgj\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d\" (UID: \"d2b114e1-248a-4a90-86a1-2490276b845d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" Oct 30 02:00:00.393417 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.393316 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9v6\" (UniqueName: \"kubernetes.io/projected/6d038f3e-e623-4011-ab10-ab444381acca-kube-api-access-cq9v6\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g\" (UID: \"6d038f3e-e623-4011-ab10-ab444381acca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" Oct 30 02:00:00.494688 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.494602 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b114e1-248a-4a90-86a1-2490276b845d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d\" (UID: \"d2b114e1-248a-4a90-86a1-2490276b845d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" Oct 30 02:00:00.494688 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.494675 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d038f3e-e623-4011-ab10-ab444381acca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g\" (UID: \"6d038f3e-e623-4011-ab10-ab444381acca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" Oct 30 02:00:00.494928 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.494739 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq4lf\" (UniqueName: \"kubernetes.io/projected/f9ef245f-d702-4a20-95b1-281f2c80f3d3-kube-api-access-wq4lf\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl\" (UID: \"f9ef245f-d702-4a20-95b1-281f2c80f3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" Oct 30 02:00:00.494928 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.494817 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ef245f-d702-4a20-95b1-281f2c80f3d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl\" (UID: \"f9ef245f-d702-4a20-95b1-281f2c80f3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" Oct 30 02:00:00.495074 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.494930 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vwgj\" (UniqueName: \"kubernetes.io/projected/d2b114e1-248a-4a90-86a1-2490276b845d-kube-api-access-4vwgj\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d\" (UID: \"d2b114e1-248a-4a90-86a1-2490276b845d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" Oct 30 02:00:00.495074 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.494969 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9v6\" (UniqueName: \"kubernetes.io/projected/6d038f3e-e623-4011-ab10-ab444381acca-kube-api-access-cq9v6\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g\" (UID: \"6d038f3e-e623-4011-ab10-ab444381acca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" Oct 30 02:00:00.495422 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.495398 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d038f3e-e623-4011-ab10-ab444381acca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g\" (UID: \"6d038f3e-e623-4011-ab10-ab444381acca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" Oct 30 02:00:00.495556 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.495532 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b114e1-248a-4a90-86a1-2490276b845d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d\" (UID: \"d2b114e1-248a-4a90-86a1-2490276b845d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" Oct 30 02:00:00.495603 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.495540 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ef245f-d702-4a20-95b1-281f2c80f3d3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl\" (UID: \"f9ef245f-d702-4a20-95b1-281f2c80f3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" Oct 30 02:00:00.518002 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.517974 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq4lf\" (UniqueName: \"kubernetes.io/projected/f9ef245f-d702-4a20-95b1-281f2c80f3d3-kube-api-access-wq4lf\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl\" (UID: \"f9ef245f-d702-4a20-95b1-281f2c80f3d3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" Oct 30 02:00:00.518321 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.518296 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vwgj\" (UniqueName: \"kubernetes.io/projected/d2b114e1-248a-4a90-86a1-2490276b845d-kube-api-access-4vwgj\") pod \"maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d\" (UID: \"d2b114e1-248a-4a90-86a1-2490276b845d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" Oct 30 02:00:00.519514 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.519484 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9v6\" (UniqueName: \"kubernetes.io/projected/6d038f3e-e623-4011-ab10-ab444381acca-kube-api-access-cq9v6\") pod \"maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g\" (UID: \"6d038f3e-e623-4011-ab10-ab444381acca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" Oct 30 02:00:00.576441 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.576404 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" Oct 30 02:00:00.618470 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.618438 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" Oct 30 02:00:00.634289 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.634260 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" Oct 30 02:00:00.920879 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.920842 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d"] Oct 30 02:00:00.961628 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:00.961492 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g"] Oct 30 02:00:00.962101 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:00:00.962076 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d038f3e_e623_4011_ab10_ab444381acca.slice/crio-ee77d467302583ca67f5693b412b0303ee7faf92bdd1a3e7e943797188b825bb WatchSource:0}: Error finding container ee77d467302583ca67f5693b412b0303ee7faf92bdd1a3e7e943797188b825bb: Status 404 returned error can't find the container with id ee77d467302583ca67f5693b412b0303ee7faf92bdd1a3e7e943797188b825bb Oct 30 02:00:01.011069 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:01.011008 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl"] Oct 30 02:00:01.065094 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:00:01.065058 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ef245f_d702_4a20_95b1_281f2c80f3d3.slice/crio-872ec914696bc673bab019c128af9562e40b45b5878ff6068903f76a5045540b WatchSource:0}: Error finding container 872ec914696bc673bab019c128af9562e40b45b5878ff6068903f76a5045540b: Status 404 returned error can't find the container with id 872ec914696bc673bab019c128af9562e40b45b5878ff6068903f76a5045540b Oct 30 02:00:01.854443 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:01.854317 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" event={"ID":"d2b114e1-248a-4a90-86a1-2490276b845d","Type":"ContainerStarted","Data":"c2c7b8d15262637e31960221e93dee8554825397de3112efebf45f657a14753c"} Oct 30 02:00:01.854443 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:01.854415 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" event={"ID":"d2b114e1-248a-4a90-86a1-2490276b845d","Type":"ContainerStarted","Data":"8460c10ca92f8d662755bec4b7aa15d99baaa3f93b92134f9243ce8dbd206bcf"} Oct 30 02:00:01.856158 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:01.856128 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" event={"ID":"6d038f3e-e623-4011-ab10-ab444381acca","Type":"ContainerStarted","Data":"895571b8343089217dedca050fe7857f84a790231f7db4b08260c1faec8c97e3"} Oct 30 02:00:01.856309 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:01.856176 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" event={"ID":"6d038f3e-e623-4011-ab10-ab444381acca","Type":"ContainerStarted","Data":"ee77d467302583ca67f5693b412b0303ee7faf92bdd1a3e7e943797188b825bb"} Oct 30 02:00:01.857871 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:01.857848 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" event={"ID":"f9ef245f-d702-4a20-95b1-281f2c80f3d3","Type":"ContainerStarted","Data":"82127188c03db91a1f9cc8c769c91a7a87340453d60e4c0e621258fa5a2c116c"} Oct 30 02:00:01.857984 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:01.857884 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" event={"ID":"f9ef245f-d702-4a20-95b1-281f2c80f3d3","Type":"ContainerStarted","Data":"872ec914696bc673bab019c128af9562e40b45b5878ff6068903f76a5045540b"} Oct 30 02:00:01.897149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:01.897081 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" podStartSLOduration=1.89706202 podStartE2EDuration="1.89706202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:01.893054571 +0000 UTC m=+2569.214011611" watchObservedRunningTime="2026-10-30 02:00:01.89706202 +0000 UTC m=+2569.218019062" Oct 30 02:00:01.943046 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:01.942986 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" podStartSLOduration=1.9429729610000002 podStartE2EDuration="1.942972961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:01.941604262 +0000 UTC m=+2569.262561302" watchObservedRunningTime="2026-10-30 02:00:01.942972961 +0000 UTC m=+2569.263930000" Oct 30 02:00:01.991626 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:01.991576 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" podStartSLOduration=1.9915598559999999 podStartE2EDuration="1.991559856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:01.986267493 +0000 UTC m=+2569.307224534" watchObservedRunningTime="2026-10-30 02:00:01.991559856 +0000 UTC m=+2569.312516924" Oct 30 02:00:02.867798 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:02.867760 5055 generic.go:356] "Generic (PLEG): container finished" podID="d2b114e1-248a-4a90-86a1-2490276b845d" containerID="c2c7b8d15262637e31960221e93dee8554825397de3112efebf45f657a14753c" exitCode=60 Oct 30 02:00:02.868283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:02.867841 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" event={"ID":"d2b114e1-248a-4a90-86a1-2490276b845d","Type":"ContainerDied","Data":"c2c7b8d15262637e31960221e93dee8554825397de3112efebf45f657a14753c"} Oct 30 02:00:02.868283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:02.868196 5055 scope.go:122] "RemoveContainer" containerID="c2c7b8d15262637e31960221e93dee8554825397de3112efebf45f657a14753c" Oct 30 02:00:02.869779 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:02.869755 5055 generic.go:356] "Generic (PLEG): container finished" podID="6d038f3e-e623-4011-ab10-ab444381acca" containerID="895571b8343089217dedca050fe7857f84a790231f7db4b08260c1faec8c97e3" exitCode=60 Oct 30 02:00:02.869865 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:02.869842 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" event={"ID":"6d038f3e-e623-4011-ab10-ab444381acca","Type":"ContainerDied","Data":"895571b8343089217dedca050fe7857f84a790231f7db4b08260c1faec8c97e3"} Oct 30 02:00:02.870292 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:02.870217 5055 scope.go:122] "RemoveContainer" containerID="895571b8343089217dedca050fe7857f84a790231f7db4b08260c1faec8c97e3" Oct 30 02:00:02.871793 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:02.871769 5055 generic.go:356] "Generic (PLEG): container finished" podID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerID="82127188c03db91a1f9cc8c769c91a7a87340453d60e4c0e621258fa5a2c116c" exitCode=60 Oct 30 02:00:02.871895 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:02.871861 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" event={"ID":"f9ef245f-d702-4a20-95b1-281f2c80f3d3","Type":"ContainerDied","Data":"82127188c03db91a1f9cc8c769c91a7a87340453d60e4c0e621258fa5a2c116c"} Oct 30 02:00:02.872092 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:02.872080 5055 scope.go:122] "RemoveContainer" containerID="82127188c03db91a1f9cc8c769c91a7a87340453d60e4c0e621258fa5a2c116c" Oct 30 02:00:03.885991 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:03.885956 5055 generic.go:356] "Generic (PLEG): container finished" podID="d2b114e1-248a-4a90-86a1-2490276b845d" containerID="56caa67b695769e5ae245286912f5441dac282e51707758cd79e042864b6041a" exitCode=60 Oct 30 02:00:03.886642 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:03.886395 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" event={"ID":"d2b114e1-248a-4a90-86a1-2490276b845d","Type":"ContainerDied","Data":"56caa67b695769e5ae245286912f5441dac282e51707758cd79e042864b6041a"} Oct 30 02:00:03.886642 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:03.886458 5055 scope.go:122] "RemoveContainer" containerID="c2c7b8d15262637e31960221e93dee8554825397de3112efebf45f657a14753c" Oct 30 02:00:03.887224 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:03.887191 5055 scope.go:122] "RemoveContainer" containerID="56caa67b695769e5ae245286912f5441dac282e51707758cd79e042864b6041a" Oct 30 02:00:03.887636 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:03.887611 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d_odh-ai-gateway-infra(d2b114e1-248a-4a90-86a1-2490276b845d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" Oct 30 02:00:03.890593 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:03.890564 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" event={"ID":"6d038f3e-e623-4011-ab10-ab444381acca","Type":"ContainerStarted","Data":"7791eaa249a9c1571d5e21fbafb7dd8d7e22ee25b5a3f025f6c22298d871cdcf"} Oct 30 02:00:03.890900 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:03.890876 5055 scope.go:122] "RemoveContainer" containerID="7791eaa249a9c1571d5e21fbafb7dd8d7e22ee25b5a3f025f6c22298d871cdcf" Oct 30 02:00:03.891251 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:03.891228 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g_odh-ai-gateway-infra(6d038f3e-e623-4011-ab10-ab444381acca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" podUID="6d038f3e-e623-4011-ab10-ab444381acca" Oct 30 02:00:03.893185 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:03.893161 5055 generic.go:356] "Generic (PLEG): container finished" podID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerID="673ab1a71f022499c758705048472e69e4be7248eacccc3bdf757e396efc0cd4" exitCode=60 Oct 30 02:00:03.893308 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:03.893247 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" event={"ID":"f9ef245f-d702-4a20-95b1-281f2c80f3d3","Type":"ContainerDied","Data":"673ab1a71f022499c758705048472e69e4be7248eacccc3bdf757e396efc0cd4"} Oct 30 02:00:03.893665 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:03.893629 5055 scope.go:122] "RemoveContainer" containerID="673ab1a71f022499c758705048472e69e4be7248eacccc3bdf757e396efc0cd4" Oct 30 02:00:03.894046 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:03.894026 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl_odh-ai-gateway-infra(f9ef245f-d702-4a20-95b1-281f2c80f3d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" Oct 30 02:00:04.018377 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:04.018292 5055 scope.go:122] "RemoveContainer" containerID="82127188c03db91a1f9cc8c769c91a7a87340453d60e4c0e621258fa5a2c116c" Oct 30 02:00:04.904825 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:04.904792 5055 scope.go:122] "RemoveContainer" containerID="56caa67b695769e5ae245286912f5441dac282e51707758cd79e042864b6041a" Oct 30 02:00:04.905314 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:04.905200 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d_odh-ai-gateway-infra(d2b114e1-248a-4a90-86a1-2490276b845d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" Oct 30 02:00:04.906741 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:04.906712 5055 generic.go:356] "Generic (PLEG): container finished" podID="6d038f3e-e623-4011-ab10-ab444381acca" containerID="7791eaa249a9c1571d5e21fbafb7dd8d7e22ee25b5a3f025f6c22298d871cdcf" exitCode=60 Oct 30 02:00:04.906892 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:04.906830 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" event={"ID":"6d038f3e-e623-4011-ab10-ab444381acca","Type":"ContainerDied","Data":"7791eaa249a9c1571d5e21fbafb7dd8d7e22ee25b5a3f025f6c22298d871cdcf"} Oct 30 02:00:04.906960 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:04.906899 5055 scope.go:122] "RemoveContainer" containerID="895571b8343089217dedca050fe7857f84a790231f7db4b08260c1faec8c97e3" Oct 30 02:00:04.907109 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:04.907086 5055 scope.go:122] "RemoveContainer" containerID="7791eaa249a9c1571d5e21fbafb7dd8d7e22ee25b5a3f025f6c22298d871cdcf" Oct 30 02:00:04.907527 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:04.907502 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g_odh-ai-gateway-infra(6d038f3e-e623-4011-ab10-ab444381acca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" podUID="6d038f3e-e623-4011-ab10-ab444381acca" Oct 30 02:00:04.910980 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:04.910959 5055 scope.go:122] "RemoveContainer" containerID="673ab1a71f022499c758705048472e69e4be7248eacccc3bdf757e396efc0cd4" Oct 30 02:00:04.911319 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:04.911300 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl_odh-ai-gateway-infra(f9ef245f-d702-4a20-95b1-281f2c80f3d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" Oct 30 02:00:05.922071 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:05.922040 5055 scope.go:122] "RemoveContainer" containerID="7791eaa249a9c1571d5e21fbafb7dd8d7e22ee25b5a3f025f6c22298d871cdcf" Oct 30 02:00:05.922626 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:05.922489 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g_odh-ai-gateway-infra(6d038f3e-e623-4011-ab10-ab444381acca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" podUID="6d038f3e-e623-4011-ab10-ab444381acca" Oct 30 02:00:08.675000 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.674972 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm"] Oct 30 02:00:08.707350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.704808 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:08.707546 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.707363 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm"] Oct 30 02:00:08.723614 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.723582 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:00:08.855255 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.854670 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa8ca13f-fr9wm\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:08.855255 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.854976 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7svd\" (UniqueName: \"kubernetes.io/projected/8c39a062-aca6-412c-b8c9-2046ef014738-kube-api-access-v7svd\") pod \"maas-api-revoke-sub-aa8ca13f-fr9wm\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:08.855255 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.855100 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa8ca13f-fr9wm\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:08.957009 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.956972 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7svd\" (UniqueName: \"kubernetes.io/projected/8c39a062-aca6-412c-b8c9-2046ef014738-kube-api-access-v7svd\") pod \"maas-api-revoke-sub-aa8ca13f-fr9wm\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:08.957219 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.957121 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa8ca13f-fr9wm\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:08.957305 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.957285 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa8ca13f-fr9wm\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:08.957799 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.957779 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa8ca13f-fr9wm\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:08.960063 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.960037 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa8ca13f-fr9wm\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:08.977788 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:08.977755 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7svd\" (UniqueName: \"kubernetes.io/projected/8c39a062-aca6-412c-b8c9-2046ef014738-kube-api-access-v7svd\") pod \"maas-api-revoke-sub-aa8ca13f-fr9wm\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:09.138508 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:09.138471 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:09.484349 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:00:09.470934 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c39a062_aca6_412c_b8c9_2046ef014738.slice/crio-8ccf638e85e044e66453f1872effad69f357360e68efe2f6d92234d9ff0c0e21 WatchSource:0}: Error finding container 8ccf638e85e044e66453f1872effad69f357360e68efe2f6d92234d9ff0c0e21: Status 404 returned error can't find the container with id 8ccf638e85e044e66453f1872effad69f357360e68efe2f6d92234d9ff0c0e21 Oct 30 02:00:09.484349 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:09.475456 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm"] Oct 30 02:00:09.994858 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:09.994818 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" event={"ID":"8c39a062-aca6-412c-b8c9-2046ef014738","Type":"ContainerStarted","Data":"6c701efca545c24c85e7310cdb98889b68745e34a5070c7cdf774f7d796ea0d0"} Oct 30 02:00:09.995382 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:09.994883 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" event={"ID":"8c39a062-aca6-412c-b8c9-2046ef014738","Type":"ContainerStarted","Data":"8ccf638e85e044e66453f1872effad69f357360e68efe2f6d92234d9ff0c0e21"} Oct 30 02:00:10.035642 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:10.035531 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" podStartSLOduration=2.035512626 podStartE2EDuration="2.035512626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:10.033030442 +0000 UTC m=+2577.353987482" watchObservedRunningTime="2026-10-30 02:00:10.035512626 +0000 UTC m=+2577.356469668" Oct 30 02:00:11.004458 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:11.004420 5055 generic.go:356] "Generic (PLEG): container finished" podID="8c39a062-aca6-412c-b8c9-2046ef014738" containerID="6c701efca545c24c85e7310cdb98889b68745e34a5070c7cdf774f7d796ea0d0" exitCode=0 Oct 30 02:00:11.004963 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:11.004516 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" event={"ID":"8c39a062-aca6-412c-b8c9-2046ef014738","Type":"ContainerDied","Data":"6c701efca545c24c85e7310cdb98889b68745e34a5070c7cdf774f7d796ea0d0"} Oct 30 02:00:12.221753 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:12.221724 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:12.315541 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:12.315432 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-xks-ca\") pod \"8c39a062-aca6-412c-b8c9-2046ef014738\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " Oct 30 02:00:12.315541 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:12.315494 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c39a062-aca6-412c-b8c9-2046ef014738-kube-api-access-v7svd\" (UniqueName: \"kubernetes.io/projected/8c39a062-aca6-412c-b8c9-2046ef014738-kube-api-access-v7svd\") pod \"8c39a062-aca6-412c-b8c9-2046ef014738\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " Oct 30 02:00:12.315779 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:12.315674 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-openshift-ca\") pod \"8c39a062-aca6-412c-b8c9-2046ef014738\" (UID: \"8c39a062-aca6-412c-b8c9-2046ef014738\") " Oct 30 02:00:12.316676 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:12.316093 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-openshift-ca" pod "8c39a062-aca6-412c-b8c9-2046ef014738" (UID: "8c39a062-aca6-412c-b8c9-2046ef014738"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:12.316676 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:12.316668 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-openshift-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:00:12.318155 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:12.318110 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-xks-ca" pod "8c39a062-aca6-412c-b8c9-2046ef014738" (UID: "8c39a062-aca6-412c-b8c9-2046ef014738"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:00:12.318507 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:12.318478 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c39a062-aca6-412c-b8c9-2046ef014738-kube-api-access-v7svd" pod "8c39a062-aca6-412c-b8c9-2046ef014738" (UID: "8c39a062-aca6-412c-b8c9-2046ef014738"). InnerVolumeSpecName "kube-api-access-v7svd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:12.417898 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:12.417861 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c39a062-aca6-412c-b8c9-2046ef014738-maas-api-xks-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:00:12.417898 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:12.417892 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7svd\" (UniqueName: \"kubernetes.io/projected/8c39a062-aca6-412c-b8c9-2046ef014738-kube-api-access-v7svd\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:00:13.024130 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:13.024101 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" Oct 30 02:00:13.024309 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:13.024142 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm" event={"ID":"8c39a062-aca6-412c-b8c9-2046ef014738","Type":"ContainerDied","Data":"8ccf638e85e044e66453f1872effad69f357360e68efe2f6d92234d9ff0c0e21"} Oct 30 02:00:13.024309 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:13.024192 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ccf638e85e044e66453f1872effad69f357360e68efe2f6d92234d9ff0c0e21" Oct 30 02:00:16.812631 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:16.812600 5055 scope.go:122] "RemoveContainer" containerID="673ab1a71f022499c758705048472e69e4be7248eacccc3bdf757e396efc0cd4" Oct 30 02:00:17.817382 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:17.812835 5055 scope.go:122] "RemoveContainer" containerID="7791eaa249a9c1571d5e21fbafb7dd8d7e22ee25b5a3f025f6c22298d871cdcf" Oct 30 02:00:18.066894 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:18.066798 5055 generic.go:356] "Generic (PLEG): container finished" podID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerID="5cc3e5b7fd28f3e4c2107b73d5aa066b0f5f51c3582c7f593502e4154af61a33" exitCode=60 Oct 30 02:00:18.066894 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:18.066868 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" event={"ID":"f9ef245f-d702-4a20-95b1-281f2c80f3d3","Type":"ContainerDied","Data":"5cc3e5b7fd28f3e4c2107b73d5aa066b0f5f51c3582c7f593502e4154af61a33"} Oct 30 02:00:18.067116 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:18.066946 5055 scope.go:122] "RemoveContainer" containerID="673ab1a71f022499c758705048472e69e4be7248eacccc3bdf757e396efc0cd4" Oct 30 02:00:18.067479 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:18.067458 5055 scope.go:122] "RemoveContainer" containerID="5cc3e5b7fd28f3e4c2107b73d5aa066b0f5f51c3582c7f593502e4154af61a33" Oct 30 02:00:18.067861 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:18.067837 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl_odh-ai-gateway-infra(f9ef245f-d702-4a20-95b1-281f2c80f3d3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" Oct 30 02:00:18.813443 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:18.813407 5055 scope.go:122] "RemoveContainer" containerID="56caa67b695769e5ae245286912f5441dac282e51707758cd79e042864b6041a" Oct 30 02:00:19.078430 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.078394 5055 generic.go:356] "Generic (PLEG): container finished" podID="6d038f3e-e623-4011-ab10-ab444381acca" containerID="8660da7a3e609ee70bce19f308f1662607e3f9896ee3d3c50c93f16554b689f2" exitCode=60 Oct 30 02:00:19.078843 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.078577 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" event={"ID":"6d038f3e-e623-4011-ab10-ab444381acca","Type":"ContainerDied","Data":"8660da7a3e609ee70bce19f308f1662607e3f9896ee3d3c50c93f16554b689f2"} Oct 30 02:00:19.078843 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.078643 5055 scope.go:122] "RemoveContainer" containerID="7791eaa249a9c1571d5e21fbafb7dd8d7e22ee25b5a3f025f6c22298d871cdcf" Oct 30 02:00:19.079129 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.079109 5055 scope.go:122] "RemoveContainer" containerID="8660da7a3e609ee70bce19f308f1662607e3f9896ee3d3c50c93f16554b689f2" Oct 30 02:00:19.079614 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:19.079590 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g_odh-ai-gateway-infra(6d038f3e-e623-4011-ab10-ab444381acca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" podUID="6d038f3e-e623-4011-ab10-ab444381acca" Oct 30 02:00:19.120647 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.120595 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl"] Oct 30 02:00:19.245427 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:19.245378 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:00:19.334723 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.334659 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" Oct 30 02:00:19.350167 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.350138 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ef245f-d702-4a20-95b1-281f2c80f3d3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ef245f-d702-4a20-95b1-281f2c80f3d3-maas-api-service-ca\") pod \"f9ef245f-d702-4a20-95b1-281f2c80f3d3\" (UID: \"f9ef245f-d702-4a20-95b1-281f2c80f3d3\") " Oct 30 02:00:19.350408 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.350393 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ef245f-d702-4a20-95b1-281f2c80f3d3-kube-api-access-wq4lf\" (UniqueName: \"kubernetes.io/projected/f9ef245f-d702-4a20-95b1-281f2c80f3d3-kube-api-access-wq4lf\") pod \"f9ef245f-d702-4a20-95b1-281f2c80f3d3\" (UID: \"f9ef245f-d702-4a20-95b1-281f2c80f3d3\") " Oct 30 02:00:19.350500 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.350479 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ef245f-d702-4a20-95b1-281f2c80f3d3-maas-api-service-ca" pod "f9ef245f-d702-4a20-95b1-281f2c80f3d3" (UID: "f9ef245f-d702-4a20-95b1-281f2c80f3d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:19.350812 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.350788 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ef245f-d702-4a20-95b1-281f2c80f3d3-maas-api-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:00:19.352773 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.352748 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ef245f-d702-4a20-95b1-281f2c80f3d3-kube-api-access-wq4lf" pod "f9ef245f-d702-4a20-95b1-281f2c80f3d3" (UID: "f9ef245f-d702-4a20-95b1-281f2c80f3d3"). InnerVolumeSpecName "kube-api-access-wq4lf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:19.452294 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:19.452251 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq4lf\" (UniqueName: \"kubernetes.io/projected/f9ef245f-d702-4a20-95b1-281f2c80f3d3-kube-api-access-wq4lf\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:00:20.093019 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.092985 5055 generic.go:356] "Generic (PLEG): container finished" podID="d2b114e1-248a-4a90-86a1-2490276b845d" containerID="4833463d4ec5459464142dd654a013d230a86c34ad335d22f9201bad2d5fe824" exitCode=60 Oct 30 02:00:20.093523 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.093066 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" event={"ID":"d2b114e1-248a-4a90-86a1-2490276b845d","Type":"ContainerDied","Data":"4833463d4ec5459464142dd654a013d230a86c34ad335d22f9201bad2d5fe824"} Oct 30 02:00:20.093523 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.093126 5055 scope.go:122] "RemoveContainer" containerID="56caa67b695769e5ae245286912f5441dac282e51707758cd79e042864b6041a" Oct 30 02:00:20.093693 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.093589 5055 scope.go:122] "RemoveContainer" containerID="4833463d4ec5459464142dd654a013d230a86c34ad335d22f9201bad2d5fe824" Oct 30 02:00:20.094445 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:20.093987 5055 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d_odh-ai-gateway-infra(d2b114e1-248a-4a90-86a1-2490276b845d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" Oct 30 02:00:20.098288 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.098263 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" Oct 30 02:00:20.098418 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.098301 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl" event={"ID":"f9ef245f-d702-4a20-95b1-281f2c80f3d3","Type":"ContainerDied","Data":"872ec914696bc673bab019c128af9562e40b45b5878ff6068903f76a5045540b"} Oct 30 02:00:20.108513 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.108490 5055 scope.go:122] "RemoveContainer" containerID="5cc3e5b7fd28f3e4c2107b73d5aa066b0f5f51c3582c7f593502e4154af61a33" Oct 30 02:00:20.192352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.165372 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g"] Oct 30 02:00:20.318693 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.316409 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl"] Oct 30 02:00:20.336834 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.329407 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888760-qtgcl"] Oct 30 02:00:20.550032 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.550006 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" Oct 30 02:00:20.582498 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.582418 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d038f3e-e623-4011-ab10-ab444381acca-kube-api-access-cq9v6\" (UniqueName: \"kubernetes.io/projected/6d038f3e-e623-4011-ab10-ab444381acca-kube-api-access-cq9v6\") pod \"6d038f3e-e623-4011-ab10-ab444381acca\" (UID: \"6d038f3e-e623-4011-ab10-ab444381acca\") " Oct 30 02:00:20.582498 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.582475 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d038f3e-e623-4011-ab10-ab444381acca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d038f3e-e623-4011-ab10-ab444381acca-maas-api-service-ca\") pod \"6d038f3e-e623-4011-ab10-ab444381acca\" (UID: \"6d038f3e-e623-4011-ab10-ab444381acca\") " Oct 30 02:00:20.583679 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.583649 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d038f3e-e623-4011-ab10-ab444381acca-maas-api-service-ca" pod "6d038f3e-e623-4011-ab10-ab444381acca" (UID: "6d038f3e-e623-4011-ab10-ab444381acca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:20.585827 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.585789 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d038f3e-e623-4011-ab10-ab444381acca-kube-api-access-cq9v6" pod "6d038f3e-e623-4011-ab10-ab444381acca" (UID: "6d038f3e-e623-4011-ab10-ab444381acca"). InnerVolumeSpecName "kube-api-access-cq9v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:20.685108 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.685066 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9v6\" (UniqueName: \"kubernetes.io/projected/6d038f3e-e623-4011-ab10-ab444381acca-kube-api-access-cq9v6\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:00:20.699520 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.699477 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d038f3e-e623-4011-ab10-ab444381acca-maas-api-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:00:20.831226 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:20.831191 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" path="/var/lib/kubelet/pods/f9ef245f-d702-4a20-95b1-281f2c80f3d3/volumes" Oct 30 02:00:21.115403 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.115073 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" event={"ID":"6d038f3e-e623-4011-ab10-ab444381acca","Type":"ContainerDied","Data":"ee77d467302583ca67f5693b412b0303ee7faf92bdd1a3e7e943797188b825bb"} Oct 30 02:00:21.115403 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.115142 5055 scope.go:122] "RemoveContainer" containerID="8660da7a3e609ee70bce19f308f1662607e3f9896ee3d3c50c93f16554b689f2" Oct 30 02:00:21.115403 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.115243 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g" Oct 30 02:00:21.151361 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.151288 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d"] Oct 30 02:00:21.182352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.180309 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g"] Oct 30 02:00:21.202396 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.193115 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-803472-29888760-vh76g"] Oct 30 02:00:21.412166 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.412138 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" Oct 30 02:00:21.524938 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.524889 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b114e1-248a-4a90-86a1-2490276b845d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b114e1-248a-4a90-86a1-2490276b845d-maas-api-service-ca\") pod \"d2b114e1-248a-4a90-86a1-2490276b845d\" (UID: \"d2b114e1-248a-4a90-86a1-2490276b845d\") " Oct 30 02:00:21.525176 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.525152 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b114e1-248a-4a90-86a1-2490276b845d-kube-api-access-4vwgj\" (UniqueName: \"kubernetes.io/projected/d2b114e1-248a-4a90-86a1-2490276b845d-kube-api-access-4vwgj\") pod \"d2b114e1-248a-4a90-86a1-2490276b845d\" (UID: \"d2b114e1-248a-4a90-86a1-2490276b845d\") " Oct 30 02:00:21.525430 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.525408 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b114e1-248a-4a90-86a1-2490276b845d-maas-api-service-ca" pod "d2b114e1-248a-4a90-86a1-2490276b845d" (UID: "d2b114e1-248a-4a90-86a1-2490276b845d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:21.525857 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.525830 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b114e1-248a-4a90-86a1-2490276b845d-maas-api-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:00:21.527915 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.527884 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b114e1-248a-4a90-86a1-2490276b845d-kube-api-access-4vwgj" pod "d2b114e1-248a-4a90-86a1-2490276b845d" (UID: "d2b114e1-248a-4a90-86a1-2490276b845d"). InnerVolumeSpecName "kube-api-access-4vwgj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:21.627631 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:21.627541 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vwgj\" (UniqueName: \"kubernetes.io/projected/d2b114e1-248a-4a90-86a1-2490276b845d-kube-api-access-4vwgj\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.142364 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:22.138844 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" event={"ID":"d2b114e1-248a-4a90-86a1-2490276b845d","Type":"ContainerDied","Data":"8460c10ca92f8d662755bec4b7aa15d99baaa3f93b92134f9243ce8dbd206bcf"} Oct 30 02:00:22.142364 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:22.138922 5055 scope.go:122] "RemoveContainer" containerID="4833463d4ec5459464142dd654a013d230a86c34ad335d22f9201bad2d5fe824" Oct 30 02:00:22.142364 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:22.139056 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d" Oct 30 02:00:22.207652 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:22.207607 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d"] Oct 30 02:00:22.213253 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:22.213221 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f1edaf-29888760-5lx9d"] Oct 30 02:00:22.826480 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:22.826441 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d038f3e-e623-4011-ab10-ab444381acca" path="/var/lib/kubelet/pods/6d038f3e-e623-4011-ab10-ab444381acca/volumes" Oct 30 02:00:22.827057 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:00:22.827034 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" path="/var/lib/kubelet/pods/d2b114e1-248a-4a90-86a1-2490276b845d/volumes" Oct 30 02:00:49.222950 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:00:49.222914 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:01:19.225684 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:01:19.225640 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:01:29.724375 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.724260 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw"] Oct 30 02:01:29.730841 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.730812 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6d038f3e-e623-4011-ab10-ab444381acca" containerName="cleanup" Oct 30 02:01:29.731078 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.731061 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d038f3e-e623-4011-ab10-ab444381acca" containerName="cleanup" Oct 30 02:01:29.731214 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.731203 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" containerName="cleanup" Oct 30 02:01:29.731320 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.731307 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" containerName="cleanup" Oct 30 02:01:29.731452 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.731439 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" containerName="cleanup" Oct 30 02:01:29.731606 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.731594 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" containerName="cleanup" Oct 30 02:01:29.731720 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.731710 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerName="cleanup" Oct 30 02:01:29.731815 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.731804 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerName="cleanup" Oct 30 02:01:29.731937 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.731926 5055 cpu_manager.go:414] "containerMap: removing container" podUID="8c39a062-aca6-412c-b8c9-2046ef014738" containerName="revoke-keys" Oct 30 02:01:29.732037 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.732026 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c39a062-aca6-412c-b8c9-2046ef014738" containerName="revoke-keys" Oct 30 02:01:29.732138 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.732128 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6d038f3e-e623-4011-ab10-ab444381acca" containerName="cleanup" Oct 30 02:01:29.732233 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.732222 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d038f3e-e623-4011-ab10-ab444381acca" containerName="cleanup" Oct 30 02:01:29.732366 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.732355 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerName="cleanup" Oct 30 02:01:29.732484 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.732471 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerName="cleanup" Oct 30 02:01:29.732649 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.732638 5055 cpu_manager.go:414] "containerMap: removing container" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerName="cleanup" Oct 30 02:01:29.732750 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.732739 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerName="cleanup" Oct 30 02:01:29.733107 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.733093 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerName="cleanup" Oct 30 02:01:29.733241 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.733232 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" containerName="cleanup" Oct 30 02:01:29.733417 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.733406 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerName="cleanup" Oct 30 02:01:29.733530 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.733520 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d038f3e-e623-4011-ab10-ab444381acca" containerName="cleanup" Oct 30 02:01:29.733635 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.733623 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d038f3e-e623-4011-ab10-ab444381acca" containerName="cleanup" Oct 30 02:01:29.733749 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.733739 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" containerName="cleanup" Oct 30 02:01:29.733853 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.733843 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c39a062-aca6-412c-b8c9-2046ef014738" containerName="revoke-keys" Oct 30 02:01:29.741160 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.741134 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.743634 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.743606 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw"] Oct 30 02:01:29.774199 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.774174 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-44rm5\"" Oct 30 02:01:29.774640 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.774606 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 02:01:29.774764 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.774667 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 02:01:29.774764 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.774738 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 02:01:29.852352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.850080 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfltq\" (UniqueName: \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-kube-api-access-qfltq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.852352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.850173 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.852352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.850200 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.852352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.850420 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.852352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.850472 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5dcb4b-ae72-4a14-b628-7178ec901366-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.852352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.850502 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.852352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.850552 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.952131 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.952082 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfltq\" (UniqueName: \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-kube-api-access-qfltq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.952344 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.952165 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.952344 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.952192 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.952344 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.952287 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.952538 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.952319 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5dcb4b-ae72-4a14-b628-7178ec901366-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.952538 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.952383 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.952538 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.952434 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.953359 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.953315 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.955541 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.955513 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.956359 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.955641 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.956359 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.955691 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.956359 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.955733 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5dcb4b-ae72-4a14-b628-7178ec901366-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.973076 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.973049 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:29.973222 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:29.973156 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfltq\" (UniqueName: \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-kube-api-access-qfltq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-k7scw\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:30.102407 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:30.098663 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:30.636280 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:01:30.636229 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5dcb4b_ae72_4a14_b628_7178ec901366.slice/crio-0afcf293da179067584975789856bd6c29cbb59d271ce7abfafa99182dafb414 WatchSource:0}: Error finding container 0afcf293da179067584975789856bd6c29cbb59d271ce7abfafa99182dafb414: Status 404 returned error can't find the container with id 0afcf293da179067584975789856bd6c29cbb59d271ce7abfafa99182dafb414 Oct 30 02:01:30.637279 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:30.637249 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw"] Oct 30 02:01:31.037213 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:31.037176 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" event={"ID":"0b5dcb4b-ae72-4a14-b628-7178ec901366","Type":"ContainerStarted","Data":"0afcf293da179067584975789856bd6c29cbb59d271ce7abfafa99182dafb414"} Oct 30 02:01:34.488699 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:34.488656 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 02:01:34.489084 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:34.488748 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 02:01:35.114228 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:35.114186 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" event={"ID":"0b5dcb4b-ae72-4a14-b628-7178ec901366","Type":"ContainerStarted","Data":"47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b"} Oct 30 02:01:35.115868 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:35.115841 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:35.118444 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:35.118416 5055 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-k7scw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:01:35.118582 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:35.118472 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" podUID="0b5dcb4b-ae72-4a14-b628-7178ec901366" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:01:35.187659 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:35.187561 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" podStartSLOduration=2.339703229 podStartE2EDuration="6.187496653s" totalImagesPullingTime="3.847793424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:35.171014116 +0000 UTC m=+2662.491971167" watchObservedRunningTime="2026-10-30 02:01:35.187496653 +0000 UTC m=+2662.508453699" Oct 30 02:01:36.124055 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:01:36.124028 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:01:49.225369 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:01:49.225292 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:02:13.345875 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:02:13.345846 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:02:13.366373 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:02:13.362926 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:02:19.241725 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:02:19.241636 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:02:49.222646 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:02:49.222608 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:03:19.251541 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:03:19.251499 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:03:49.221314 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:03:49.221274 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:04:19.221747 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:04:19.221708 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:04:22.528711 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:22.528633 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd"] Oct 30 02:04:22.529212 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:22.528879 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" containerID="cri-o://6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8" gracePeriod=30 Oct 30 02:04:25.231077 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:25.231037 5055 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:04:25.231536 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:25.231106 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:04:27.832551 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.832525 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 02:04:27.887632 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.887596 5055 generic.go:356] "Generic (PLEG): container finished" podID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerID="6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8" exitCode=0 Oct 30 02:04:27.887831 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.887687 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" event={"ID":"c04f7dba-9b73-448b-9c01-c92970e4f715","Type":"ContainerDied","Data":"6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8"} Oct 30 02:04:27.887831 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.887809 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" Oct 30 02:04:27.887997 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.887948 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd" event={"ID":"c04f7dba-9b73-448b-9c01-c92970e4f715","Type":"ContainerDied","Data":"1ab4a68ecd9d6585458f7c8a2cbd9d9efcb056ed3249ffd3814ef1660e635e73"} Oct 30 02:04:27.888061 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.888012 5055 scope.go:122] "RemoveContainer" containerID="6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8" Oct 30 02:04:27.895930 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.895900 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-data\") pod \"c04f7dba-9b73-448b-9c01-c92970e4f715\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " Oct 30 02:04:27.896074 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.896039 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-token\" (UniqueName: \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-token\") pod \"c04f7dba-9b73-448b-9c01-c92970e4f715\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " Oct 30 02:04:27.896139 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.896112 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-podinfo\") pod \"c04f7dba-9b73-448b-9c01-c92970e4f715\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " Oct 30 02:04:27.896225 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.896207 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04f7dba-9b73-448b-9c01-c92970e4f715-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04f7dba-9b73-448b-9c01-c92970e4f715-istiod-ca-cert\") pod \"c04f7dba-9b73-448b-9c01-c92970e4f715\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " Oct 30 02:04:27.896288 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.896267 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-kube-api-access-j46n4\" (UniqueName: \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-kube-api-access-j46n4\") pod \"c04f7dba-9b73-448b-9c01-c92970e4f715\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " Oct 30 02:04:27.896374 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.896316 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-certs\") pod \"c04f7dba-9b73-448b-9c01-c92970e4f715\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " Oct 30 02:04:27.896460 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.896362 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-data" pod "c04f7dba-9b73-448b-9c01-c92970e4f715" (UID: "c04f7dba-9b73-448b-9c01-c92970e4f715"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:27.896460 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.896428 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-credential-socket\") pod \"c04f7dba-9b73-448b-9c01-c92970e4f715\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " Oct 30 02:04:27.896565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.896485 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-socket\") pod \"c04f7dba-9b73-448b-9c01-c92970e4f715\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " Oct 30 02:04:27.896640 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.896603 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-envoy\") pod \"c04f7dba-9b73-448b-9c01-c92970e4f715\" (UID: \"c04f7dba-9b73-448b-9c01-c92970e4f715\") " Oct 30 02:04:27.897025 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.897001 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04f7dba-9b73-448b-9c01-c92970e4f715-istiod-ca-cert" pod "c04f7dba-9b73-448b-9c01-c92970e4f715" (UID: "c04f7dba-9b73-448b-9c01-c92970e4f715"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:04:27.897472 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.897453 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-credential-socket" pod "c04f7dba-9b73-448b-9c01-c92970e4f715" (UID: "c04f7dba-9b73-448b-9c01-c92970e4f715"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:27.897598 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.897571 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:04:27.897700 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.897688 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c04f7dba-9b73-448b-9c01-c92970e4f715-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:04:27.898122 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.898100 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-socket" pod "c04f7dba-9b73-448b-9c01-c92970e4f715" (UID: "c04f7dba-9b73-448b-9c01-c92970e4f715"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:27.901263 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.901005 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-envoy" pod "c04f7dba-9b73-448b-9c01-c92970e4f715" (UID: "c04f7dba-9b73-448b-9c01-c92970e4f715"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:27.902632 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.902581 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-certs" pod "c04f7dba-9b73-448b-9c01-c92970e4f715" (UID: "c04f7dba-9b73-448b-9c01-c92970e4f715"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:04:27.903131 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.903104 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-podinfo" pod "c04f7dba-9b73-448b-9c01-c92970e4f715" (UID: "c04f7dba-9b73-448b-9c01-c92970e4f715"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:04:27.903263 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.903221 5055 scope.go:122] "RemoveContainer" containerID="6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8" Oct 30 02:04:27.903483 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.903450 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-token" pod "c04f7dba-9b73-448b-9c01-c92970e4f715" (UID: "c04f7dba-9b73-448b-9c01-c92970e4f715"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:27.906320 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:04:27.904486 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8\": container with ID starting with 6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8 not found: ID does not exist" containerID="6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8" Oct 30 02:04:27.906320 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.904553 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8"} err="failed to get container status \"6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8\": rpc error: code = NotFound desc = could not find container \"6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8\": container with ID starting with 6f4a6bace81e3775a8b10ef82595565fe9968d3aaf0613c284623c3de1d536d8 not found: ID does not exist" Oct 30 02:04:27.911921 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.911892 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-kube-api-access-j46n4" pod "c04f7dba-9b73-448b-9c01-c92970e4f715" (UID: "c04f7dba-9b73-448b-9c01-c92970e4f715"). InnerVolumeSpecName "kube-api-access-j46n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:04:27.999052 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.999014 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:04:27.999052 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.999044 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:04:27.999052 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.999054 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j46n4\" (UniqueName: \"kubernetes.io/projected/c04f7dba-9b73-448b-9c01-c92970e4f715-kube-api-access-j46n4\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:04:27.999052 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.999065 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:04:27.999429 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.999074 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:04:27.999429 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.999082 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:04:27.999429 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:27.999105 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c04f7dba-9b73-448b-9c01-c92970e4f715-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:04:28.238792 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:28.238757 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd"] Oct 30 02:04:28.251360 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:28.251315 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-jb2vd"] Oct 30 02:04:28.822681 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:04:28.822643 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" path="/var/lib/kubelet/pods/c04f7dba-9b73-448b-9c01-c92970e4f715/volumes" Oct 30 02:04:49.226668 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:04:49.226627 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:05:13.036182 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:05:13.036135 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm"] Oct 30 02:05:13.048683 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:05:13.048644 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa8ca13f-fr9wm"] Oct 30 02:05:14.824264 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:05:14.824229 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c39a062-aca6-412c-b8c9-2046ef014738" path="/var/lib/kubelet/pods/8c39a062-aca6-412c-b8c9-2046ef014738/volumes" Oct 30 02:05:19.222459 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:05:19.222417 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:05:49.222184 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:05:49.222148 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:06:15.093469 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:15.093431 5055 scope.go:122] "RemoveContainer" containerID="6c701efca545c24c85e7310cdb98889b68745e34a5070c7cdf774f7d796ea0d0" Oct 30 02:06:19.220496 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:06:19.220460 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:06:32.951437 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:32.951378 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2"] Oct 30 02:06:32.952030 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:32.951634 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" containerID="cri-o://5f0e97e36159149a41343bc88d6810352d504743a5464ad911351bd01c0e70c7" gracePeriod=30 Oct 30 02:06:35.619266 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:35.619224 5055 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:06:35.619722 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:35.619287 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:06:38.319616 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.319527 5055 generic.go:356] "Generic (PLEG): container finished" podID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerID="5f0e97e36159149a41343bc88d6810352d504743a5464ad911351bd01c0e70c7" exitCode=0 Oct 30 02:06:38.319616 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.319566 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" event={"ID":"8b57a425-bc92-4a04-a643-eb6ae9cc7b44","Type":"ContainerDied","Data":"5f0e97e36159149a41343bc88d6810352d504743a5464ad911351bd01c0e70c7"} Oct 30 02:06:38.320056 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.319636 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" event={"ID":"8b57a425-bc92-4a04-a643-eb6ae9cc7b44","Type":"ContainerDied","Data":"ba108e1fdcdbd49966ed73cb978b941abcd1afbdb57a2ff9bd102f199fe05625"} Oct 30 02:06:38.320056 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.319662 5055 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba108e1fdcdbd49966ed73cb978b941abcd1afbdb57a2ff9bd102f199fe05625" Oct 30 02:06:38.340854 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.340814 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 02:06:38.360971 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.360921 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-envoy\") pod \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " Oct 30 02:06:38.361154 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.360985 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-podinfo\") pod \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " Oct 30 02:06:38.361154 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.361019 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-kube-api-access-6qlhl\" (UniqueName: \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-kube-api-access-6qlhl\") pod \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " Oct 30 02:06:38.361154 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.361064 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istiod-ca-cert\") pod \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " Oct 30 02:06:38.361154 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.361103 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-data\") pod \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " Oct 30 02:06:38.361406 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.361231 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-socket\") pod \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " Oct 30 02:06:38.361406 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.361307 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-credential-socket\") pod \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " Oct 30 02:06:38.361530 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.361513 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-certs\") pod \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " Oct 30 02:06:38.361655 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.361639 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-token\" (UniqueName: \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-token\") pod \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\" (UID: \"8b57a425-bc92-4a04-a643-eb6ae9cc7b44\") " Oct 30 02:06:38.361709 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.361657 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-data" pod "8b57a425-bc92-4a04-a643-eb6ae9cc7b44" (UID: "8b57a425-bc92-4a04-a643-eb6ae9cc7b44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:38.362257 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.362235 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.362584 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.362559 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-credential-socket" pod "8b57a425-bc92-4a04-a643-eb6ae9cc7b44" (UID: "8b57a425-bc92-4a04-a643-eb6ae9cc7b44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:38.362720 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.362703 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-socket" pod "8b57a425-bc92-4a04-a643-eb6ae9cc7b44" (UID: "8b57a425-bc92-4a04-a643-eb6ae9cc7b44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:38.363275 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.363249 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-certs" pod "8b57a425-bc92-4a04-a643-eb6ae9cc7b44" (UID: "8b57a425-bc92-4a04-a643-eb6ae9cc7b44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:38.364781 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.364757 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istiod-ca-cert" pod "8b57a425-bc92-4a04-a643-eb6ae9cc7b44" (UID: "8b57a425-bc92-4a04-a643-eb6ae9cc7b44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:38.368469 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.368437 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-envoy" pod "8b57a425-bc92-4a04-a643-eb6ae9cc7b44" (UID: "8b57a425-bc92-4a04-a643-eb6ae9cc7b44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:38.376623 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.374245 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-podinfo" pod "8b57a425-bc92-4a04-a643-eb6ae9cc7b44" (UID: "8b57a425-bc92-4a04-a643-eb6ae9cc7b44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:06:38.376623 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.376539 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-token" pod "8b57a425-bc92-4a04-a643-eb6ae9cc7b44" (UID: "8b57a425-bc92-4a04-a643-eb6ae9cc7b44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:38.377497 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.377473 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-kube-api-access-6qlhl" pod "8b57a425-bc92-4a04-a643-eb6ae9cc7b44" (UID: "8b57a425-bc92-4a04-a643-eb6ae9cc7b44"). InnerVolumeSpecName "kube-api-access-6qlhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:38.479353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.478187 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.479353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.478229 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.479353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.478247 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.479353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.478260 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.479353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.478274 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.479353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.478287 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.479353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.478300 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qlhl\" (UniqueName: \"kubernetes.io/projected/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-kube-api-access-6qlhl\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.479353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.478315 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b57a425-bc92-4a04-a643-eb6ae9cc7b44-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:38.980685 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.980650 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r"] Oct 30 02:06:38.980901 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:38.980842 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerName="istio-proxy" containerID="cri-o://7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2" gracePeriod=30 Oct 30 02:06:39.331347 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:39.331278 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2" Oct 30 02:06:39.392786 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:39.392748 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2"] Oct 30 02:06:39.409120 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:39.409080 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-6bdg2"] Oct 30 02:06:40.821932 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:40.821894 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" path="/var/lib/kubelet/pods/8b57a425-bc92-4a04-a643-eb6ae9cc7b44/volumes" Oct 30 02:06:44.355398 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.355367 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 02:06:44.371942 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.371895 5055 generic.go:356] "Generic (PLEG): container finished" podID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerID="7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2" exitCode=0 Oct 30 02:06:44.372120 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.372002 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" Oct 30 02:06:44.372120 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.372044 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" event={"ID":"101455c3-027f-4c48-b8d9-6691e5bb3f1f","Type":"ContainerDied","Data":"7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2"} Oct 30 02:06:44.372199 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.372145 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r" event={"ID":"101455c3-027f-4c48-b8d9-6691e5bb3f1f","Type":"ContainerDied","Data":"59b716df81d548b60643dfba9635bbe8d65d0b79b3c22577a04ba8a3ff3f43ff"} Oct 30 02:06:44.372199 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.372177 5055 scope.go:122] "RemoveContainer" containerID="7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2" Oct 30 02:06:44.387722 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.387694 5055 scope.go:122] "RemoveContainer" containerID="7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2" Oct 30 02:06:44.388122 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:06:44.388094 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2\": container with ID starting with 7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2 not found: ID does not exist" containerID="7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2" Oct 30 02:06:44.388248 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.388130 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2"} err="failed to get container status \"7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2\": rpc error: code = NotFound desc = could not find container \"7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2\": container with ID starting with 7b4af76a0be61e4f3e49eae5017be3ce7f56007c8f808ac2d2bdd700260d84b2 not found: ID does not exist" Oct 30 02:06:44.434385 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.434358 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-kube-api-access-ch9xh\" (UniqueName: \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-kube-api-access-ch9xh\") pod \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " Oct 30 02:06:44.434641 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.434620 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istiod-ca-cert\") pod \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " Oct 30 02:06:44.435085 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.435045 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istiod-ca-cert" pod "101455c3-027f-4c48-b8d9-6691e5bb3f1f" (UID: "101455c3-027f-4c48-b8d9-6691e5bb3f1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:44.435203 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.435168 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-envoy\") pod \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " Oct 30 02:06:44.435438 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.435415 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-socket\") pod \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " Oct 30 02:06:44.435836 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.435778 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-socket" pod "101455c3-027f-4c48-b8d9-6691e5bb3f1f" (UID: "101455c3-027f-4c48-b8d9-6691e5bb3f1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.435941 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.435923 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-certs\") pod \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " Oct 30 02:06:44.436040 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.436019 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-token\" (UniqueName: \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-token\") pod \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " Oct 30 02:06:44.436557 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.436193 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-podinfo\") pod \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " Oct 30 02:06:44.436557 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.436319 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-credential-socket\") pod \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " Oct 30 02:06:44.436557 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.436378 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-data\") pod \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\" (UID: \"101455c3-027f-4c48-b8d9-6691e5bb3f1f\") " Oct 30 02:06:44.436557 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.436208 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-certs" pod "101455c3-027f-4c48-b8d9-6691e5bb3f1f" (UID: "101455c3-027f-4c48-b8d9-6691e5bb3f1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.437283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.436767 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-credential-socket" pod "101455c3-027f-4c48-b8d9-6691e5bb3f1f" (UID: "101455c3-027f-4c48-b8d9-6691e5bb3f1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.437283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.436894 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-data" pod "101455c3-027f-4c48-b8d9-6691e5bb3f1f" (UID: "101455c3-027f-4c48-b8d9-6691e5bb3f1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.437283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.437095 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.437283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.437109 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.437283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.437119 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.437283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.437127 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.437283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.437138 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.462601 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.462560 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-envoy" pod "101455c3-027f-4c48-b8d9-6691e5bb3f1f" (UID: "101455c3-027f-4c48-b8d9-6691e5bb3f1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.462601 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.462594 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-podinfo" pod "101455c3-027f-4c48-b8d9-6691e5bb3f1f" (UID: "101455c3-027f-4c48-b8d9-6691e5bb3f1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:06:44.462808 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.462708 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-kube-api-access-ch9xh" pod "101455c3-027f-4c48-b8d9-6691e5bb3f1f" (UID: "101455c3-027f-4c48-b8d9-6691e5bb3f1f"). InnerVolumeSpecName "kube-api-access-ch9xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:44.462868 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.462852 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-token" pod "101455c3-027f-4c48-b8d9-6691e5bb3f1f" (UID: "101455c3-027f-4c48-b8d9-6691e5bb3f1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:44.539013 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.538967 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.539013 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.539015 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.539237 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.539031 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch9xh\" (UniqueName: \"kubernetes.io/projected/101455c3-027f-4c48-b8d9-6691e5bb3f1f-kube-api-access-ch9xh\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.539237 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.539050 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/101455c3-027f-4c48-b8d9-6691e5bb3f1f-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.743620 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.743583 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r"] Oct 30 02:06:44.760932 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.760892 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-jrp8r"] Oct 30 02:06:44.843675 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:44.843638 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" path="/var/lib/kubelet/pods/101455c3-027f-4c48-b8d9-6691e5bb3f1f/volumes" Oct 30 02:06:45.676726 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:45.676658 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw"] Oct 30 02:06:45.677241 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:45.676889 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" podUID="0b5dcb4b-ae72-4a14-b628-7178ec901366" containerName="discovery" containerID="cri-o://47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b" gracePeriod=30 Oct 30 02:06:46.006200 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.006173 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:06:46.174913 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.174881 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-kubeconfig\") pod \"0b5dcb4b-ae72-4a14-b628-7178ec901366\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " Oct 30 02:06:46.175164 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.174929 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-token\" (UniqueName: \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-token\") pod \"0b5dcb4b-ae72-4a14-b628-7178ec901366\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " Oct 30 02:06:46.175164 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.175011 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-cacerts\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-cacerts\") pod \"0b5dcb4b-ae72-4a14-b628-7178ec901366\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " Oct 30 02:06:46.175164 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.175061 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-ca-configmap\") pod \"0b5dcb4b-ae72-4a14-b628-7178ec901366\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " Oct 30 02:06:46.175164 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.175106 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b5dcb4b-ae72-4a14-b628-7178ec901366-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5dcb4b-ae72-4a14-b628-7178ec901366-local-certs\") pod \"0b5dcb4b-ae72-4a14-b628-7178ec901366\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " Oct 30 02:06:46.175439 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.175357 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-dns-cert\") pod \"0b5dcb4b-ae72-4a14-b628-7178ec901366\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " Oct 30 02:06:46.175501 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.175463 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-kube-api-access-qfltq\" (UniqueName: \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-kube-api-access-qfltq\") pod \"0b5dcb4b-ae72-4a14-b628-7178ec901366\" (UID: \"0b5dcb4b-ae72-4a14-b628-7178ec901366\") " Oct 30 02:06:46.175643 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.175619 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-ca-configmap" pod "0b5dcb4b-ae72-4a14-b628-7178ec901366" (UID: "0b5dcb4b-ae72-4a14-b628-7178ec901366"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:46.176308 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.176244 5055 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-ca-configmap\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.182091 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.182030 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-dns-cert" pod "0b5dcb4b-ae72-4a14-b628-7178ec901366" (UID: "0b5dcb4b-ae72-4a14-b628-7178ec901366"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:46.182268 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.182249 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-kube-api-access-qfltq" pod "0b5dcb4b-ae72-4a14-b628-7178ec901366" (UID: "0b5dcb4b-ae72-4a14-b628-7178ec901366"). InnerVolumeSpecName "kube-api-access-qfltq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:46.182469 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.182427 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-cacerts" pod "0b5dcb4b-ae72-4a14-b628-7178ec901366" (UID: "0b5dcb4b-ae72-4a14-b628-7178ec901366"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:46.184136 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.183557 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-kubeconfig" pod "0b5dcb4b-ae72-4a14-b628-7178ec901366" (UID: "0b5dcb4b-ae72-4a14-b628-7178ec901366"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:06:46.184136 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.183991 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b5dcb4b-ae72-4a14-b628-7178ec901366-local-certs" pod "0b5dcb4b-ae72-4a14-b628-7178ec901366" (UID: "0b5dcb4b-ae72-4a14-b628-7178ec901366"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:46.184363 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.184240 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-token" pod "0b5dcb4b-ae72-4a14-b628-7178ec901366" (UID: "0b5dcb4b-ae72-4a14-b628-7178ec901366"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:46.277626 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.277518 5055 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-kubeconfig\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.277626 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.277564 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.277626 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.277578 5055 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-cacerts\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.277626 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.277589 5055 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b5dcb4b-ae72-4a14-b628-7178ec901366-local-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.277626 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.277601 5055 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0b5dcb4b-ae72-4a14-b628-7178ec901366-istio-csr-dns-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.277626 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.277614 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfltq\" (UniqueName: \"kubernetes.io/projected/0b5dcb4b-ae72-4a14-b628-7178ec901366-kube-api-access-qfltq\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:06:46.391148 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.391109 5055 generic.go:356] "Generic (PLEG): container finished" podID="0b5dcb4b-ae72-4a14-b628-7178ec901366" containerID="47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b" exitCode=0 Oct 30 02:06:46.391409 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.391389 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" event={"ID":"0b5dcb4b-ae72-4a14-b628-7178ec901366","Type":"ContainerDied","Data":"47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b"} Oct 30 02:06:46.391483 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.391444 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" event={"ID":"0b5dcb4b-ae72-4a14-b628-7178ec901366","Type":"ContainerDied","Data":"0afcf293da179067584975789856bd6c29cbb59d271ce7abfafa99182dafb414"} Oct 30 02:06:46.391483 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.391470 5055 scope.go:122] "RemoveContainer" containerID="47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b" Oct 30 02:06:46.391590 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.391579 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw" Oct 30 02:06:46.405591 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.405566 5055 scope.go:122] "RemoveContainer" containerID="47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b" Oct 30 02:06:46.405959 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:06:46.405939 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b\": container with ID starting with 47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b not found: ID does not exist" containerID="47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b" Oct 30 02:06:46.406044 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.405970 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b"} err="failed to get container status \"47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b\": rpc error: code = NotFound desc = could not find container \"47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b\": container with ID starting with 47c3917c95fe180451ec6dd743fc23396d3ba8cb1479612bb0d168bc77376e7b not found: ID does not exist" Oct 30 02:06:46.450175 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.450140 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw"] Oct 30 02:06:46.455151 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.455118 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-k7scw"] Oct 30 02:06:46.823496 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:46.823461 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5dcb4b-ae72-4a14-b628-7178ec901366" path="/var/lib/kubelet/pods/0b5dcb4b-ae72-4a14-b628-7178ec901366/volumes" Oct 30 02:06:48.443937 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.443902 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5"] Oct 30 02:06:48.444645 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.444623 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" containerName="cleanup" Oct 30 02:06:48.444859 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.444830 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" containerName="cleanup" Oct 30 02:06:48.444972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.444903 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0b5dcb4b-ae72-4a14-b628-7178ec901366" containerName="discovery" Oct 30 02:06:48.444972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.444916 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5dcb4b-ae72-4a14-b628-7178ec901366" containerName="discovery" Oct 30 02:06:48.444972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.444932 5055 cpu_manager.go:414] "containerMap: removing container" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerName="istio-proxy" Oct 30 02:06:48.444972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.444940 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerName="istio-proxy" Oct 30 02:06:48.444972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.444957 5055 cpu_manager.go:414] "containerMap: removing container" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" Oct 30 02:06:48.444972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.444966 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" Oct 30 02:06:48.445203 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445030 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" Oct 30 02:06:48.445203 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445037 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" Oct 30 02:06:48.445203 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445078 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6d038f3e-e623-4011-ab10-ab444381acca" containerName="cleanup" Oct 30 02:06:48.445203 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445086 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d038f3e-e623-4011-ab10-ab444381acca" containerName="cleanup" Oct 30 02:06:48.445397 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445258 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b114e1-248a-4a90-86a1-2490276b845d" containerName="cleanup" Oct 30 02:06:48.445397 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445272 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5dcb4b-ae72-4a14-b628-7178ec901366" containerName="discovery" Oct 30 02:06:48.445397 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445283 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="101455c3-027f-4c48-b8d9-6691e5bb3f1f" containerName="istio-proxy" Oct 30 02:06:48.445397 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445292 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b57a425-bc92-4a04-a643-eb6ae9cc7b44" containerName="istio-proxy" Oct 30 02:06:48.445397 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445307 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ef245f-d702-4a20-95b1-281f2c80f3d3" containerName="cleanup" Oct 30 02:06:48.445397 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445319 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04f7dba-9b73-448b-9c01-c92970e4f715" containerName="istio-proxy" Oct 30 02:06:48.445397 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.445385 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d038f3e-e623-4011-ab10-ab444381acca" containerName="cleanup" Oct 30 02:06:48.449645 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.449615 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:06:48.466303 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.466266 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5"] Oct 30 02:06:48.472079 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.472037 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f89697-dockercfg-nzsjc\"" Oct 30 02:06:48.508817 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.508785 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f89697\"" Oct 30 02:06:48.613349 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.595998 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1019d4-0665-4023-b5fe-52f5f2394723-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5\" (UID: \"0f1019d4-0665-4023-b5fe-52f5f2394723\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:06:48.613349 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.596110 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk4zn\" (UniqueName: \"kubernetes.io/projected/0f1019d4-0665-4023-b5fe-52f5f2394723-kube-api-access-lk4zn\") pod \"payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5\" (UID: \"0f1019d4-0665-4023-b5fe-52f5f2394723\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:06:48.697992 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.697873 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1019d4-0665-4023-b5fe-52f5f2394723-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5\" (UID: \"0f1019d4-0665-4023-b5fe-52f5f2394723\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:06:48.698305 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.698285 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk4zn\" (UniqueName: \"kubernetes.io/projected/0f1019d4-0665-4023-b5fe-52f5f2394723-kube-api-access-lk4zn\") pod \"payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5\" (UID: \"0f1019d4-0665-4023-b5fe-52f5f2394723\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:06:48.699610 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.699584 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1019d4-0665-4023-b5fe-52f5f2394723-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5\" (UID: \"0f1019d4-0665-4023-b5fe-52f5f2394723\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:06:48.721112 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.720999 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk4zn\" (UniqueName: \"kubernetes.io/projected/0f1019d4-0665-4023-b5fe-52f5f2394723-kube-api-access-lk4zn\") pod \"payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5\" (UID: \"0f1019d4-0665-4023-b5fe-52f5f2394723\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:06:48.767273 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:48.767244 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:06:49.113972 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:06:49.089970 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1019d4_0665_4023_b5fe_52f5f2394723.slice/crio-af295f337a8bf1854b7f983f27d29f3413c021323789dcca1e2fec610d5e8ec7 WatchSource:0}: Error finding container af295f337a8bf1854b7f983f27d29f3413c021323789dcca1e2fec610d5e8ec7: Status 404 returned error can't find the container with id af295f337a8bf1854b7f983f27d29f3413c021323789dcca1e2fec610d5e8ec7 Oct 30 02:06:49.133671 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:49.129418 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5"] Oct 30 02:06:49.227962 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:06:49.227923 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:06:49.455647 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:49.455606 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" event={"ID":"0f1019d4-0665-4023-b5fe-52f5f2394723","Type":"ContainerStarted","Data":"0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00"} Oct 30 02:06:49.456049 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:49.455666 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" event={"ID":"0f1019d4-0665-4023-b5fe-52f5f2394723","Type":"ContainerStarted","Data":"af295f337a8bf1854b7f983f27d29f3413c021323789dcca1e2fec610d5e8ec7"} Oct 30 02:06:49.456049 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:49.455690 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:06:49.505717 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:49.505654 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" podStartSLOduration=1.5056346729999999 podStartE2EDuration="1.505634673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:49.500143038 +0000 UTC m=+2976.821100079" watchObservedRunningTime="2026-10-30 02:06:49.505634673 +0000 UTC m=+2976.826591714" Oct 30 02:06:59.271372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.271307 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7"] Oct 30 02:06:59.275162 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.275131 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.291994 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.291946 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f81908-serving-cert\"" Oct 30 02:06:59.310304 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.310264 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7"] Oct 30 02:06:59.377451 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.377411 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-metrics-certs\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.377654 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.377535 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-maas-api-tls\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.377654 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.377629 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k92m7\" (UniqueName: \"kubernetes.io/projected/902d35cf-e127-4756-9a4b-ac9225465e17-kube-api-access-k92m7\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.377758 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.377744 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/902d35cf-e127-4756-9a4b-ac9225465e17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.479451 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.479413 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-metrics-certs\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.479637 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.479496 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-maas-api-tls\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.479637 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.479594 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k92m7\" (UniqueName: \"kubernetes.io/projected/902d35cf-e127-4756-9a4b-ac9225465e17-kube-api-access-k92m7\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.479784 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.479715 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/902d35cf-e127-4756-9a4b-ac9225465e17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.480537 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.480504 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/902d35cf-e127-4756-9a4b-ac9225465e17-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.482904 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.482877 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-metrics-certs\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.486666 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.486640 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-maas-api-tls\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.500778 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.500750 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k92m7\" (UniqueName: \"kubernetes.io/projected/902d35cf-e127-4756-9a4b-ac9225465e17-kube-api-access-k92m7\") pod \"maas-api-e2e-inf-b-f81908-86d586699-mrqg7\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.599632 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.599532 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:06:59.909857 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.909754 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7"] Oct 30 02:06:59.919769 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:06:59.919741 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902d35cf_e127_4756_9a4b_ac9225465e17.slice/crio-554af6e375f4a1b5983a23e74778dd4e176bfe564f57ffb8c608eeaa0471718d WatchSource:0}: Error finding container 554af6e375f4a1b5983a23e74778dd4e176bfe564f57ffb8c608eeaa0471718d: Status 404 returned error can't find the container with id 554af6e375f4a1b5983a23e74778dd4e176bfe564f57ffb8c608eeaa0471718d Oct 30 02:06:59.921472 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:06:59.921449 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:07:00.490785 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:00.490694 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:07:00.575372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:00.574175 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" event={"ID":"902d35cf-e127-4756-9a4b-ac9225465e17","Type":"ContainerStarted","Data":"61a78128eb479988e0ccbf0ade276082e26f1d833afa26db453d42424c939a02"} Oct 30 02:07:00.575372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:00.574250 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" event={"ID":"902d35cf-e127-4756-9a4b-ac9225465e17","Type":"ContainerStarted","Data":"554af6e375f4a1b5983a23e74778dd4e176bfe564f57ffb8c608eeaa0471718d"} Oct 30 02:07:00.575372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:00.575301 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:07:00.620078 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:00.620023 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" podStartSLOduration=1.305494637 podStartE2EDuration="1.620005956s" totalImagesPullingTime="314.511319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:00.617355764 +0000 UTC m=+2987.938312805" watchObservedRunningTime="2026-10-30 02:07:00.620005956 +0000 UTC m=+2987.940963060" Oct 30 02:07:07.597731 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:07.597696 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:07:13.418698 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:13.418669 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:07:13.419689 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:13.419668 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:07:15.122938 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:15.122906 5055 scope.go:122] "RemoveContainer" containerID="5f0e97e36159149a41343bc88d6810352d504743a5464ad911351bd01c0e70c7" Oct 30 02:07:19.225423 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:07:19.225364 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:07:33.615451 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.615351 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5"] Oct 30 02:07:33.621765 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.621738 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.640496 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.640461 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f89697-serving-cert\"" Oct 30 02:07:33.644748 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.644717 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5"] Oct 30 02:07:33.709401 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.709345 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-metrics-certs\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.709692 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.709665 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9vw5\" (UniqueName: \"kubernetes.io/projected/679f8047-1bb0-4123-92d7-6474c85ce2be-kube-api-access-c9vw5\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.709848 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.709714 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-maas-api-tls\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.709848 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.709748 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679f8047-1bb0-4123-92d7-6474c85ce2be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.812067 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.812024 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-metrics-certs\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.814373 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.814344 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9vw5\" (UniqueName: \"kubernetes.io/projected/679f8047-1bb0-4123-92d7-6474c85ce2be-kube-api-access-c9vw5\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.815011 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.814772 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-maas-api-tls\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.817460 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.815174 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679f8047-1bb0-4123-92d7-6474c85ce2be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.817643 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.815889 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679f8047-1bb0-4123-92d7-6474c85ce2be-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.817742 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:07:33.814982 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-serving-cert: secret "maas-api-e2e-inf-a-f89697-serving-cert" not found Oct 30 02:07:33.817814 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:07:33.817739 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-maas-api-tls podName:679f8047-1bb0-4123-92d7-6474c85ce2be nodeName:}" failed. No retries permitted until 2026-10-30 02:07:34.317715559 +0000 UTC m=+3021.638672581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-maas-api-tls") pod "maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" (UID: "679f8047-1bb0-4123-92d7-6474c85ce2be") : secret "maas-api-e2e-inf-a-f89697-serving-cert" not found Oct 30 02:07:33.824992 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.824969 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-metrics-certs\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:33.843640 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:33.843608 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9vw5\" (UniqueName: \"kubernetes.io/projected/679f8047-1bb0-4123-92d7-6474c85ce2be-kube-api-access-c9vw5\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:34.329167 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:34.329116 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-maas-api-tls\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:34.344407 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:34.343024 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-maas-api-tls\") pod \"maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:34.540124 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:34.540085 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:34.951151 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:34.951053 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5"] Oct 30 02:07:35.977969 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:35.977937 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" event={"ID":"679f8047-1bb0-4123-92d7-6474c85ce2be","Type":"ContainerStarted","Data":"9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc"} Oct 30 02:07:35.978565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:35.977988 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" event={"ID":"679f8047-1bb0-4123-92d7-6474c85ce2be","Type":"ContainerStarted","Data":"30b7fb1b6efc984399271099c6b10b57a3d33a3c9ec5f7633a5afba2e5eefe30"} Oct 30 02:07:35.978565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:35.978033 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:36.030787 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:36.030726 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" podStartSLOduration=2.617945224 podStartE2EDuration="3.030709573s" totalImagesPullingTime="412.764349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:36.025443428 +0000 UTC m=+3023.346400468" watchObservedRunningTime="2026-10-30 02:07:36.030709573 +0000 UTC m=+3023.351666612" Oct 30 02:07:40.058366 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.054404 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk"] Oct 30 02:07:40.166024 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.165987 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk"] Oct 30 02:07:40.166265 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.166138 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.182456 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.182388 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-aaf045-serving-cert\"" Oct 30 02:07:40.316872 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.316749 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43fbb343-2521-4767-b229-0912ab4880d1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.317086 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.316945 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-metrics-certs\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.317086 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.317065 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-maas-api-tls\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.317217 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.317167 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frh6b\" (UniqueName: \"kubernetes.io/projected/43fbb343-2521-4767-b229-0912ab4880d1-kube-api-access-frh6b\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.418345 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.418295 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43fbb343-2521-4767-b229-0912ab4880d1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.418567 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.418397 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-metrics-certs\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.418567 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.418462 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-maas-api-tls\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.418567 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.418534 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frh6b\" (UniqueName: \"kubernetes.io/projected/43fbb343-2521-4767-b229-0912ab4880d1-kube-api-access-frh6b\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.419117 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.419086 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43fbb343-2521-4767-b229-0912ab4880d1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.421414 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.421390 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-maas-api-tls\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.421485 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.421390 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-metrics-certs\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.439174 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.439138 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frh6b\" (UniqueName: \"kubernetes.io/projected/43fbb343-2521-4767-b229-0912ab4880d1-kube-api-access-frh6b\") pod \"maas-api-e2e-inf-a-aaf045-8589d7b975-695bk\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.489902 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.489864 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:40.827546 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:40.827522 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk"] Oct 30 02:07:40.830904 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:07:40.830873 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43fbb343_2521_4767_b229_0912ab4880d1.slice/crio-956e242c6b7d4039d8bb940fe49b6697e21d01768184459a46aed0c632ba3f88 WatchSource:0}: Error finding container 956e242c6b7d4039d8bb940fe49b6697e21d01768184459a46aed0c632ba3f88: Status 404 returned error can't find the container with id 956e242c6b7d4039d8bb940fe49b6697e21d01768184459a46aed0c632ba3f88 Oct 30 02:07:41.149656 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:41.149619 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" event={"ID":"43fbb343-2521-4767-b229-0912ab4880d1","Type":"ContainerStarted","Data":"956e242c6b7d4039d8bb940fe49b6697e21d01768184459a46aed0c632ba3f88"} Oct 30 02:07:41.996221 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:41.996192 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:07:42.166060 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:42.166014 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" event={"ID":"43fbb343-2521-4767-b229-0912ab4880d1","Type":"ContainerStarted","Data":"ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765"} Oct 30 02:07:42.166518 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:42.166086 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:42.215311 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:42.215259 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" podStartSLOduration=2.922375716 podStartE2EDuration="3.215240219s" totalImagesPullingTime="292.864503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:42.212926945 +0000 UTC m=+3029.533883987" watchObservedRunningTime="2026-10-30 02:07:42.215240219 +0000 UTC m=+3029.536197260" Oct 30 02:07:48.182548 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:48.182516 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:07:49.221662 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:07:49.221618 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:07:55.209482 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.209445 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h"] Oct 30 02:07:55.217685 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.217650 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.240171 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.240135 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h"] Oct 30 02:07:55.247425 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.247394 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-717065\"/\"openshift-service-ca.crt\"" Oct 30 02:07:55.247425 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.247417 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-717065\"/\"kube-root-ca.crt\"" Oct 30 02:07:55.247615 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.247470 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-717065\"/\"default-dockercfg-65fhl\"" Oct 30 02:07:55.247615 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.247537 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-717065\"/\"test-model-717065-kserve-self-signed-certs\"" Oct 30 02:07:55.337029 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.336989 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-home\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.337243 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.337123 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tmp-dir\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.337243 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.337158 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-model-cache\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.337243 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.337178 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tls-certs\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.337243 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.337214 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-dshm\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.337518 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.337315 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-httvn\" (UniqueName: \"kubernetes.io/projected/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-kube-api-access-httvn\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.438753 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.438705 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-home\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.438972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.438775 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tmp-dir\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.438972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.438818 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-model-cache\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.438972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.438838 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tls-certs\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.438972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.438868 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-dshm\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.438972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.438896 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-httvn\" (UniqueName: \"kubernetes.io/projected/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-kube-api-access-httvn\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.439282 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.439207 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-home\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.439368 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.439274 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tmp-dir\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.439368 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.439306 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-model-cache\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.441635 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.441606 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-dshm\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.441774 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.441751 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tls-certs\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.459902 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.459825 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-httvn\" (UniqueName: \"kubernetes.io/projected/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-kube-api-access-httvn\") pod \"test-model-717065-kserve-876d96bc9-x8v6h\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.547232 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.546914 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:07:55.920801 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:55.920721 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h"] Oct 30 02:07:55.931381 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:07:55.931348 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79be7f9d_f615_4c9f_9db8_f20c0f1a6206.slice/crio-749d9205b1fc03d053567eb502804048c04f9951e76621173bd12416d5278290 WatchSource:0}: Error finding container 749d9205b1fc03d053567eb502804048c04f9951e76621173bd12416d5278290: Status 404 returned error can't find the container with id 749d9205b1fc03d053567eb502804048c04f9951e76621173bd12416d5278290 Oct 30 02:07:56.284616 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:56.284574 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" event={"ID":"79be7f9d-f615-4c9f-9db8-f20c0f1a6206","Type":"ContainerStarted","Data":"5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667"} Oct 30 02:07:56.284616 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:56.284619 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" event={"ID":"79be7f9d-f615-4c9f-9db8-f20c0f1a6206","Type":"ContainerStarted","Data":"749d9205b1fc03d053567eb502804048c04f9951e76621173bd12416d5278290"} Oct 30 02:07:56.329797 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:56.329730 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" podStartSLOduration=1.329714581 podStartE2EDuration="1.329714581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:56.328507323 +0000 UTC m=+3043.649464344" watchObservedRunningTime="2026-10-30 02:07:56.329714581 +0000 UTC m=+3043.650671621" Oct 30 02:07:56.548836 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:07:56.548760 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:08:05.550936 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:05.550892 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:08:05.661353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:05.660529 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:08:06.460400 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:06.460371 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:08:19.223137 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:08:19.223076 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:08:43.279202 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.279164 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v"] Oct 30 02:08:43.284027 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.284003 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:08:43.303521 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.303491 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-72f9d5\"/\"default-dockercfg-tm8zp\"" Oct 30 02:08:43.305710 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.305681 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v"] Oct 30 02:08:43.312575 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.312548 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-72f9d5\"/\"kube-root-ca.crt\"" Oct 30 02:08:43.316049 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.316020 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-extmod-shr-user-72f9d5\"/\"openshift-service-ca.crt\"" Oct 30 02:08:43.368687 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.368650 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dqpp\" (UniqueName: \"kubernetes.io/projected/49dc7504-c3b9-42da-bfcc-78a51040b9c2-kube-api-access-7dqpp\") pod \"e2e-usage-sim-2bb30d-6b56449588-pj79v\" (UID: \"49dc7504-c3b9-42da-bfcc-78a51040b9c2\") " pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:08:43.368923 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.368722 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/49dc7504-c3b9-42da-bfcc-78a51040b9c2-tls\") pod \"e2e-usage-sim-2bb30d-6b56449588-pj79v\" (UID: \"49dc7504-c3b9-42da-bfcc-78a51040b9c2\") " pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:08:43.404063 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.404029 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-extmod-shr-user-72f9d5\"/\"e2e-usage-sim-2bb30d-cert\"" Oct 30 02:08:43.471162 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.471123 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dqpp\" (UniqueName: \"kubernetes.io/projected/49dc7504-c3b9-42da-bfcc-78a51040b9c2-kube-api-access-7dqpp\") pod \"e2e-usage-sim-2bb30d-6b56449588-pj79v\" (UID: \"49dc7504-c3b9-42da-bfcc-78a51040b9c2\") " pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:08:43.471375 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.471202 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/49dc7504-c3b9-42da-bfcc-78a51040b9c2-tls\") pod \"e2e-usage-sim-2bb30d-6b56449588-pj79v\" (UID: \"49dc7504-c3b9-42da-bfcc-78a51040b9c2\") " pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:08:43.474269 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.474236 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/49dc7504-c3b9-42da-bfcc-78a51040b9c2-tls\") pod \"e2e-usage-sim-2bb30d-6b56449588-pj79v\" (UID: \"49dc7504-c3b9-42da-bfcc-78a51040b9c2\") " pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:08:43.491856 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.491825 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dqpp\" (UniqueName: \"kubernetes.io/projected/49dc7504-c3b9-42da-bfcc-78a51040b9c2-kube-api-access-7dqpp\") pod \"e2e-usage-sim-2bb30d-6b56449588-pj79v\" (UID: \"49dc7504-c3b9-42da-bfcc-78a51040b9c2\") " pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:08:43.602057 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.601949 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:08:43.921075 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.920986 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v"] Oct 30 02:08:43.931636 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:08:43.931605 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49dc7504_c3b9_42da_bfcc_78a51040b9c2.slice/crio-fd23384c206404e43c08210c85e106a1465d343f4e9cfe2a96af791f99bfe0bc WatchSource:0}: Error finding container fd23384c206404e43c08210c85e106a1465d343f4e9cfe2a96af791f99bfe0bc: Status 404 returned error can't find the container with id fd23384c206404e43c08210c85e106a1465d343f4e9cfe2a96af791f99bfe0bc Oct 30 02:08:43.960722 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:43.960691 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" event={"ID":"49dc7504-c3b9-42da-bfcc-78a51040b9c2","Type":"ContainerStarted","Data":"fd23384c206404e43c08210c85e106a1465d343f4e9cfe2a96af791f99bfe0bc"} Oct 30 02:08:44.972065 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:44.972020 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" event={"ID":"49dc7504-c3b9-42da-bfcc-78a51040b9c2","Type":"ContainerStarted","Data":"10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0"} Oct 30 02:08:44.972623 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:44.972271 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:08:44.979219 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:44.979192 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:08:45.015236 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:08:45.015180 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" podStartSLOduration=2.015166283 podStartE2EDuration="2.015166283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:45.013285442 +0000 UTC m=+3092.334242483" watchObservedRunningTime="2026-10-30 02:08:45.015166283 +0000 UTC m=+3092.336123342" Oct 30 02:08:49.258364 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:08:49.256303 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:09:19.247815 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:09:19.247775 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:09:27.086107 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.086072 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2"] Oct 30 02:09:27.091625 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.091600 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.116376 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.116150 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-601d4b-serving-cert\"" Oct 30 02:09:27.120243 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.120214 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2"] Oct 30 02:09:27.150647 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.150613 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6cbk\" (UniqueName: \"kubernetes.io/projected/dec73c66-91e7-4f40-9a01-0080db07ec87-kube-api-access-d6cbk\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.150854 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.150675 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-maas-api-tls\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.150955 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.150927 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec73c66-91e7-4f40-9a01-0080db07ec87-openshift-service-ca\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.151095 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.151080 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-metrics-certs\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.252842 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.252802 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-metrics-certs\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.253044 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.252970 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6cbk\" (UniqueName: \"kubernetes.io/projected/dec73c66-91e7-4f40-9a01-0080db07ec87-kube-api-access-d6cbk\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.253044 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.253011 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-maas-api-tls\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.253191 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:09:27.253172 5055 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-serving-cert: secret "maas-api-e2e-inf-b-601d4b-serving-cert" not found Oct 30 02:09:27.253259 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.253177 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec73c66-91e7-4f40-9a01-0080db07ec87-openshift-service-ca\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.253259 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:09:27.253250 5055 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-maas-api-tls podName:dec73c66-91e7-4f40-9a01-0080db07ec87 nodeName:}" failed. No retries permitted until 2026-10-30 02:09:27.75323433 +0000 UTC m=+3135.074191361 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-maas-api-tls") pod "maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" (UID: "dec73c66-91e7-4f40-9a01-0080db07ec87") : secret "maas-api-e2e-inf-b-601d4b-serving-cert" not found Oct 30 02:09:27.253995 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.253969 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec73c66-91e7-4f40-9a01-0080db07ec87-openshift-service-ca\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.261400 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.261305 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-metrics-certs\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.295826 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.295792 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6cbk\" (UniqueName: \"kubernetes.io/projected/dec73c66-91e7-4f40-9a01-0080db07ec87-kube-api-access-d6cbk\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.760138 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.760098 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-maas-api-tls\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:27.762967 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:27.762939 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-maas-api-tls\") pod \"maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:28.009142 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:28.009100 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:28.352733 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:09:28.352686 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec73c66_91e7_4f40_9a01_0080db07ec87.slice/crio-b3b9864d500d2a2909a02f3caef09ddbd07928c55f048ac046c6c45f1f7b7c06 WatchSource:0}: Error finding container b3b9864d500d2a2909a02f3caef09ddbd07928c55f048ac046c6c45f1f7b7c06: Status 404 returned error can't find the container with id b3b9864d500d2a2909a02f3caef09ddbd07928c55f048ac046c6c45f1f7b7c06 Oct 30 02:09:28.353241 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:28.353216 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2"] Oct 30 02:09:28.406387 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:28.406318 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" event={"ID":"dec73c66-91e7-4f40-9a01-0080db07ec87","Type":"ContainerStarted","Data":"b3b9864d500d2a2909a02f3caef09ddbd07928c55f048ac046c6c45f1f7b7c06"} Oct 30 02:09:29.419121 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:29.419070 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" event={"ID":"dec73c66-91e7-4f40-9a01-0080db07ec87","Type":"ContainerStarted","Data":"06b1653b2aa411bc4da7df8b635dfedbe395e168cc07f80ac401eb873026079a"} Oct 30 02:09:29.419567 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:29.419165 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:29.468971 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:29.468916 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" podStartSLOduration=1.932038382 podStartE2EDuration="2.468901538s" totalImagesPullingTime="536.863156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:29.466598992 +0000 UTC m=+3136.787556032" watchObservedRunningTime="2026-10-30 02:09:29.468901538 +0000 UTC m=+3136.789858578" Oct 30 02:09:32.211115 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.211073 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln"] Oct 30 02:09:32.216290 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.216266 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.236723 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.236688 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-22e99d-serving-cert\"" Oct 30 02:09:32.256286 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.256239 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln"] Oct 30 02:09:32.330762 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.330721 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-maas-api-tls\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.330997 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.330785 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ae70a5-1283-439e-9b9d-12644878a758-openshift-service-ca\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.331062 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.331000 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkpjm\" (UniqueName: \"kubernetes.io/projected/98ae70a5-1283-439e-9b9d-12644878a758-kube-api-access-tkpjm\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.331062 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.331044 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-metrics-certs\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.432204 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.432159 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-metrics-certs\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.432489 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.432345 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-maas-api-tls\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.432489 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.432408 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ae70a5-1283-439e-9b9d-12644878a758-openshift-service-ca\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.432617 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.432600 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkpjm\" (UniqueName: \"kubernetes.io/projected/98ae70a5-1283-439e-9b9d-12644878a758-kube-api-access-tkpjm\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.433904 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.433865 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ae70a5-1283-439e-9b9d-12644878a758-openshift-service-ca\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.435559 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.435526 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-metrics-certs\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.438352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.436475 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-maas-api-tls\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.455487 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.455455 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkpjm\" (UniqueName: \"kubernetes.io/projected/98ae70a5-1283-439e-9b9d-12644878a758-kube-api-access-tkpjm\") pod \"maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.531824 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.531733 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:32.863017 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:32.862914 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln"] Oct 30 02:09:32.874655 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:09:32.874620 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ae70a5_1283_439e_9b9d_12644878a758.slice/crio-6c493ce3e3699b5895d8bd5f7cdc7ca2cc3b75fae42c79691e9e29c558abc368 WatchSource:0}: Error finding container 6c493ce3e3699b5895d8bd5f7cdc7ca2cc3b75fae42c79691e9e29c558abc368: Status 404 returned error can't find the container with id 6c493ce3e3699b5895d8bd5f7cdc7ca2cc3b75fae42c79691e9e29c558abc368 Oct 30 02:09:33.539391 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:33.539264 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" event={"ID":"98ae70a5-1283-439e-9b9d-12644878a758","Type":"ContainerStarted","Data":"576a256a2fb1c835a10c0cd736bf682423184a22ee22e55e94b8a979a904ff4c"} Oct 30 02:09:33.539884 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:33.539854 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" event={"ID":"98ae70a5-1283-439e-9b9d-12644878a758","Type":"ContainerStarted","Data":"6c493ce3e3699b5895d8bd5f7cdc7ca2cc3b75fae42c79691e9e29c558abc368"} Oct 30 02:09:33.540205 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:33.540161 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:33.586734 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:33.586662 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" podStartSLOduration=1.216579759 podStartE2EDuration="1.586646165s" totalImagesPullingTime="370.066406ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:33.583873266 +0000 UTC m=+3140.904830309" watchObservedRunningTime="2026-10-30 02:09:33.586646165 +0000 UTC m=+3140.907603249" Oct 30 02:09:35.431805 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:35.431769 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:09:39.554179 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:09:39.554148 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:09:49.223133 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:09:49.223093 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:10:00.380115 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.380076 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh"] Oct 30 02:10:00.385203 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.385180 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.429008 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.428974 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh"] Oct 30 02:10:00.444728 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.444684 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gzj2\" (UniqueName: \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-kube-api-access-7gzj2\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.444910 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.444741 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.444910 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.444834 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.444910 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.444878 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.445047 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.445029 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.445089 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.445071 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.445311 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.445288 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.445372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.445347 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.445477 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.445455 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.495253 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.495218 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-601d4b-openshift-default-dockercfg-7nw2n\"" Oct 30 02:10:00.547524 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.547481 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.547524 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.547524 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.547803 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.547588 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.547803 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.547681 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gzj2\" (UniqueName: \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-kube-api-access-7gzj2\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.547803 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.547707 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.547803 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.547777 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.548021 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.547816 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.548021 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.547906 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.548021 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.547945 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.548180 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.548067 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.548246 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.548219 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.548306 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.548289 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.548388 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.548357 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.548661 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.548637 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.550527 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.550504 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.551000 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.550976 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.577019 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.576984 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.577400 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.577374 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gzj2\" (UniqueName: \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-kube-api-access-7gzj2\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:00.707135 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:00.707103 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:01.042745 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:01.042642 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh"] Oct 30 02:10:01.053725 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:10:01.053698 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b9ecdc_93c7_4284_9c50_92ce07e34d90.slice/crio-2532d3597cab5134dce7292f4049dfd35dd13189a6dc6f65c5a47bf92fd0129d WatchSource:0}: Error finding container 2532d3597cab5134dce7292f4049dfd35dd13189a6dc6f65c5a47bf92fd0129d: Status 404 returned error can't find the container with id 2532d3597cab5134dce7292f4049dfd35dd13189a6dc6f65c5a47bf92fd0129d Oct 30 02:10:01.056671 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:01.056633 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 02:10:01.056792 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:01.056726 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 02:10:01.056792 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:01.056773 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 02:10:01.798408 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:01.798317 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" event={"ID":"a0b9ecdc-93c7-4284-9c50-92ce07e34d90","Type":"ContainerStarted","Data":"55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c"} Oct 30 02:10:01.798951 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:01.798438 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" event={"ID":"a0b9ecdc-93c7-4284-9c50-92ce07e34d90","Type":"ContainerStarted","Data":"2532d3597cab5134dce7292f4049dfd35dd13189a6dc6f65c5a47bf92fd0129d"} Oct 30 02:10:01.884363 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:01.880061 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" podStartSLOduration=1.880037392 podStartE2EDuration="1.880037392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:01.85982756 +0000 UTC m=+3169.180784600" watchObservedRunningTime="2026-10-30 02:10:01.880037392 +0000 UTC m=+3169.200994433" Oct 30 02:10:02.708235 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:02.708194 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:02.709913 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:02.709885 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 02:10:02.710068 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:02.709935 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" podUID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 02:10:03.708459 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:03.708409 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 02:10:03.709010 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:03.708496 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" podUID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 02:10:04.707874 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:04.707831 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 02:10:04.708280 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:04.708236 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" podUID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 02:10:05.707794 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:05.707754 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 02:10:05.708410 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:05.708379 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" podUID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 02:10:06.708145 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:06.708104 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 30 02:10:06.708618 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:06.708154 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" podUID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 30 02:10:07.713895 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:07.713858 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:07.714453 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:07.714432 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:07.715409 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:07.715384 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:10:19.221804 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:10:19.221761 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:10:45.493911 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.493816 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn"] Oct 30 02:10:45.499127 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.499099 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.537647 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.537613 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn"] Oct 30 02:10:45.545189 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.545151 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dfd823-2436-423a-b2f6-67a537b6a329-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.545189 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.545202 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.545496 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.545247 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.545496 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.545304 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.545496 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.545378 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.545496 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.545415 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.545496 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.545461 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.545496 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.545488 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txbwd\" (UniqueName: \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-kube-api-access-txbwd\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.545836 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.545787 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.646961 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.646920 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dfd823-2436-423a-b2f6-67a537b6a329-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.646961 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.646968 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647293 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647003 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647293 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647052 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647293 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647088 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647293 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647119 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647293 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647153 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647293 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647175 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txbwd\" (UniqueName: \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-kube-api-access-txbwd\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647657 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647392 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647657 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647552 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647657 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647601 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647819 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647777 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dfd823-2436-423a-b2f6-67a537b6a329-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647819 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647788 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.647945 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.647914 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.650434 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.650403 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.650565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.650548 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.669131 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.669099 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.670302 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.670276 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txbwd\" (UniqueName: \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-kube-api-access-txbwd\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:45.823870 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:45.823777 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:46.179319 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:10:46.179283 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0dfd823_2436_423a_b2f6_67a537b6a329.slice/crio-952de181e08d701e5a4515d731b64d7890e15beb1dba2afdb7993276862a2735 WatchSource:0}: Error finding container 952de181e08d701e5a4515d731b64d7890e15beb1dba2afdb7993276862a2735: Status 404 returned error can't find the container with id 952de181e08d701e5a4515d731b64d7890e15beb1dba2afdb7993276862a2735 Oct 30 02:10:46.182529 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:46.182497 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn"] Oct 30 02:10:46.183783 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:46.183749 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 02:10:46.183894 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:46.183821 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 02:10:46.183894 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:46.183861 5055 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"} Oct 30 02:10:46.191673 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:46.191642 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" event={"ID":"a0dfd823-2436-423a-b2f6-67a537b6a329","Type":"ContainerStarted","Data":"952de181e08d701e5a4515d731b64d7890e15beb1dba2afdb7993276862a2735"} Oct 30 02:10:47.205977 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:47.205936 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" event={"ID":"a0dfd823-2436-423a-b2f6-67a537b6a329","Type":"ContainerStarted","Data":"200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007"} Oct 30 02:10:47.271353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:47.270377 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" podStartSLOduration=2.270359089 podStartE2EDuration="2.270359089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:47.270019248 +0000 UTC m=+3214.590976289" watchObservedRunningTime="2026-10-30 02:10:47.270359089 +0000 UTC m=+3214.591316131" Oct 30 02:10:47.824926 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:47.824890 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:47.826815 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:47.826786 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 02:10:47.826971 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:47.826895 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 02:10:48.824450 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:48.824417 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 02:10:48.824871 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:48.824475 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 02:10:49.243193 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:10:49.243156 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:10:49.824745 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:49.824706 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 02:10:49.825208 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:49.824783 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 02:10:50.824666 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:50.824635 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 02:10:50.824929 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:50.824691 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 02:10:51.825007 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:51.824972 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 02:10:51.825466 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:51.825027 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 02:10:52.824172 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:52.824137 5055 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 30 02:10:52.824400 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:52.824208 5055 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 30 02:10:53.851975 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:53.851925 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:53.853553 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:53.853530 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:53.854798 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:53.854773 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:10:56.060625 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.060583 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm"] Oct 30 02:10:56.066856 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.066828 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.083786 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.083750 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-601d4b\"/\"kube-root-ca.crt\"" Oct 30 02:10:56.083959 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.083861 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-601d4b\"/\"openshift-service-ca.crt\"" Oct 30 02:10:56.085502 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.085478 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm"] Oct 30 02:10:56.096285 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.096258 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-601d4b\"/\"default-dockercfg-nptsr\"" Oct 30 02:10:56.096545 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.096463 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-601d4b\"/\"test-model-601d4b-kserve-self-signed-certs\"" Oct 30 02:10:56.197886 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.197842 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e91e96-2e4d-4df8-b9fa-928f401194c6-tls-certs\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.198276 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.198254 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-dshm\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.198503 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.198483 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kll4\" (UniqueName: \"kubernetes.io/projected/28e91e96-2e4d-4df8-b9fa-928f401194c6-kube-api-access-6kll4\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.198610 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.198593 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-home\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.198733 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.198717 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-model-cache\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.198798 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.198755 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-tmp-dir\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.300181 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.300132 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kll4\" (UniqueName: \"kubernetes.io/projected/28e91e96-2e4d-4df8-b9fa-928f401194c6-kube-api-access-6kll4\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.300405 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.300235 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-home\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.300405 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.300316 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-model-cache\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.300405 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.300363 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-tmp-dir\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.300587 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.300477 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e91e96-2e4d-4df8-b9fa-928f401194c6-tls-certs\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.300646 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.300593 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-dshm\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.300899 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.300865 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-home\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.301055 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.300908 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-tmp-dir\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.301055 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.300964 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-model-cache\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.303252 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.303226 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-dshm\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.303594 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.303576 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e91e96-2e4d-4df8-b9fa-928f401194c6-tls-certs\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.332209 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.332095 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kll4\" (UniqueName: \"kubernetes.io/projected/28e91e96-2e4d-4df8-b9fa-928f401194c6-kube-api-access-6kll4\") pod \"test-model-601d4b-kserve-688559f56f-6bcxm\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.396005 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.395694 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:56.763688 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:10:56.763656 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28e91e96_2e4d_4df8_b9fa_928f401194c6.slice/crio-dafeaacd9f188fead37e0874e94373c638c1b8350596bcff1f9d1ad35fe4e825 WatchSource:0}: Error finding container dafeaacd9f188fead37e0874e94373c638c1b8350596bcff1f9d1ad35fe4e825: Status 404 returned error can't find the container with id dafeaacd9f188fead37e0874e94373c638c1b8350596bcff1f9d1ad35fe4e825 Oct 30 02:10:56.766915 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:56.766883 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm"] Oct 30 02:10:57.300484 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:57.300434 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" event={"ID":"28e91e96-2e4d-4df8-b9fa-928f401194c6","Type":"ContainerStarted","Data":"552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1"} Oct 30 02:10:57.300950 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:57.300499 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" event={"ID":"28e91e96-2e4d-4df8-b9fa-928f401194c6","Type":"ContainerStarted","Data":"dafeaacd9f188fead37e0874e94373c638c1b8350596bcff1f9d1ad35fe4e825"} Oct 30 02:10:57.349281 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:57.349223 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" podStartSLOduration=1.349205834 podStartE2EDuration="1.349205834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:57.348292619 +0000 UTC m=+3224.669249673" watchObservedRunningTime="2026-10-30 02:10:57.349205834 +0000 UTC m=+3224.670162877" Oct 30 02:10:57.397184 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:57.397138 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:10:59.043252 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.043221 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc"] Oct 30 02:10:59.047905 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.047879 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.076663 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.076626 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc"] Oct 30 02:10:59.081177 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.081148 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-aaf045\"/\"kube-root-ca.crt\"" Oct 30 02:10:59.081439 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.081197 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-aaf045\"/\"default-dockercfg-6992q\"" Oct 30 02:10:59.081439 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.081229 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-aaf045\"/\"openshift-service-ca.crt\"" Oct 30 02:10:59.081439 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.081359 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-aaf045\"/\"test-model-aaf045-kserve-self-signed-certs\"" Oct 30 02:10:59.150012 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.149978 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-model-cache\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.150183 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.150044 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tls-certs\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.150183 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.150109 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-home\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.150183 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.150155 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-dshm\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.150311 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.150275 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbjz9\" (UniqueName: \"kubernetes.io/projected/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-kube-api-access-pbjz9\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.150368 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.150349 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tmp-dir\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.251585 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.251544 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-model-cache\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.251789 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.251621 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tls-certs\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.251789 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.251647 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-home\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.251913 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.251799 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-dshm\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.251913 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.251895 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbjz9\" (UniqueName: \"kubernetes.io/projected/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-kube-api-access-pbjz9\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.252030 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.251958 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tmp-dir\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.252030 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.252010 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-model-cache\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.252188 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.252166 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-home\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.252461 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.252382 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tmp-dir\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.254489 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.254468 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tls-certs\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.254668 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.254647 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-dshm\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.294224 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.294110 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbjz9\" (UniqueName: \"kubernetes.io/projected/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-kube-api-access-pbjz9\") pod \"test-model-aaf045-kserve-c55566bb9-jslrc\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.377684 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.377644 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:10:59.736161 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:10:59.736131 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaeba5ed_7e13_4ff1_a2b5_2ecb56ca6b4c.slice/crio-310a146a1988322a648fdef414fb385393bdac0b354617ceb0b3823ddf121c04 WatchSource:0}: Error finding container 310a146a1988322a648fdef414fb385393bdac0b354617ceb0b3823ddf121c04: Status 404 returned error can't find the container with id 310a146a1988322a648fdef414fb385393bdac0b354617ceb0b3823ddf121c04 Oct 30 02:10:59.738754 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:10:59.738713 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc"] Oct 30 02:11:00.358392 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:11:00.358351 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" event={"ID":"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c","Type":"ContainerStarted","Data":"b9ad353838f6992950c557beead1153c9e7b679e400af7797e48e58b9e78cb1c"} Oct 30 02:11:00.358909 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:11:00.358405 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" event={"ID":"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c","Type":"ContainerStarted","Data":"310a146a1988322a648fdef414fb385393bdac0b354617ceb0b3823ddf121c04"} Oct 30 02:11:00.378484 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:11:00.378451 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:11:00.406787 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:11:00.406724 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" podStartSLOduration=1.406705544 podStartE2EDuration="1.406705544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:00.401152313 +0000 UTC m=+3227.722109356" watchObservedRunningTime="2026-10-30 02:11:00.406705544 +0000 UTC m=+3227.727662588" Oct 30 02:11:06.397067 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:11:06.397019 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:11:06.411515 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:11:06.411474 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:11:06.427680 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:11:06.427652 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:11:09.378507 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:11:09.378478 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:11:09.392563 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:11:09.392536 5055 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:11:09.453126 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:11:09.453095 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:11:19.257569 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:11:19.257473 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:11:49.223215 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:11:49.223169 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:12:13.477644 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:13.477602 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:12:13.480896 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:13.480870 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:12:14.606091 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:14.606051 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h"] Oct 30 02:12:19.222073 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:12:19.222036 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:12:23.007387 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.007350 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx"] Oct 30 02:12:23.012403 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.012370 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.041746 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.041707 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-340ae6-serving-cert\"" Oct 30 02:12:23.059936 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.059903 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx"] Oct 30 02:12:23.068230 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.068193 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3337f877-9bf8-4787-b840-3a07f57c330f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.068415 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.068305 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrnzh\" (UniqueName: \"kubernetes.io/projected/3337f877-9bf8-4787-b840-3a07f57c330f-kube-api-access-wrnzh\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.068481 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.068436 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-metrics-certs\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.068481 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.068469 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-maas-api-tls\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.169792 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.169754 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrnzh\" (UniqueName: \"kubernetes.io/projected/3337f877-9bf8-4787-b840-3a07f57c330f-kube-api-access-wrnzh\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.170029 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.169836 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-metrics-certs\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.170029 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.169857 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-maas-api-tls\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.170134 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.170061 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3337f877-9bf8-4787-b840-3a07f57c330f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.170823 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.170735 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3337f877-9bf8-4787-b840-3a07f57c330f-openshift-service-ca\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.172698 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.172675 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-metrics-certs\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.172796 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.172710 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-maas-api-tls\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.189936 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.189910 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrnzh\" (UniqueName: \"kubernetes.io/projected/3337f877-9bf8-4787-b840-3a07f57c330f-kube-api-access-wrnzh\") pod \"maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.370413 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.361654 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:23.747686 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:12:23.747640 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3337f877_9bf8_4787_b840_3a07f57c330f.slice/crio-19aae18c4acd10bdd980f6f6c2061bc64875d28837be11ad3423de93deaa26e9 WatchSource:0}: Error finding container 19aae18c4acd10bdd980f6f6c2061bc64875d28837be11ad3423de93deaa26e9: Status 404 returned error can't find the container with id 19aae18c4acd10bdd980f6f6c2061bc64875d28837be11ad3423de93deaa26e9 Oct 30 02:12:23.749760 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.749733 5055 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:12:23.753254 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:23.753229 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx"] Oct 30 02:12:24.129247 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:24.126144 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" event={"ID":"3337f877-9bf8-4787-b840-3a07f57c330f","Type":"ContainerStarted","Data":"19aae18c4acd10bdd980f6f6c2061bc64875d28837be11ad3423de93deaa26e9"} Oct 30 02:12:25.136682 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:25.136649 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" event={"ID":"3337f877-9bf8-4787-b840-3a07f57c330f","Type":"ContainerStarted","Data":"be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e"} Oct 30 02:12:25.137219 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:25.136723 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:25.180249 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:25.180188 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" podStartSLOduration=2.692895219 podStartE2EDuration="3.180172212s" totalImagesPullingTime="487.276993ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:12:25.176841767 +0000 UTC m=+3312.497798808" watchObservedRunningTime="2026-10-30 02:12:25.180172212 +0000 UTC m=+3312.501129284" Oct 30 02:12:29.651093 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:29.651032 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" podUID="79be7f9d-f615-4c9f-9db8-f20c0f1a6206" containerName="main" containerID="cri-o://5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667" gracePeriod=45 Oct 30 02:12:29.975554 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:29.975528 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:12:30.009651 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.009614 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-kube-api-access-httvn\" (UniqueName: \"kubernetes.io/projected/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-kube-api-access-httvn\") pod \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " Oct 30 02:12:30.009876 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.009769 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tmp-dir\") pod \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " Oct 30 02:12:30.009876 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.009842 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-home\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-home\") pod \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " Oct 30 02:12:30.009996 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.009882 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-model-cache\") pod \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " Oct 30 02:12:30.009996 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.009983 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tls-certs\" (UniqueName: \"kubernetes.io/secret/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tls-certs\") pod \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " Oct 30 02:12:30.010104 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.010022 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-dshm\") pod \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\" (UID: \"79be7f9d-f615-4c9f-9db8-f20c0f1a6206\") " Oct 30 02:12:30.010158 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.010115 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tmp-dir" pod "79be7f9d-f615-4c9f-9db8-f20c0f1a6206" (UID: "79be7f9d-f615-4c9f-9db8-f20c0f1a6206"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:30.010218 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.010153 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-home" pod "79be7f9d-f615-4c9f-9db8-f20c0f1a6206" (UID: "79be7f9d-f615-4c9f-9db8-f20c0f1a6206"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:30.010218 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.010199 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-model-cache" pod "79be7f9d-f615-4c9f-9db8-f20c0f1a6206" (UID: "79be7f9d-f615-4c9f-9db8-f20c0f1a6206"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:30.010689 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.010629 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:12:30.010689 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.010654 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:12:30.010689 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.010667 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:12:30.012777 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.012753 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-dshm" pod "79be7f9d-f615-4c9f-9db8-f20c0f1a6206" (UID: "79be7f9d-f615-4c9f-9db8-f20c0f1a6206"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:30.012913 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.012868 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tls-certs" pod "79be7f9d-f615-4c9f-9db8-f20c0f1a6206" (UID: "79be7f9d-f615-4c9f-9db8-f20c0f1a6206"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:30.013169 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.013151 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-kube-api-access-httvn" pod "79be7f9d-f615-4c9f-9db8-f20c0f1a6206" (UID: "79be7f9d-f615-4c9f-9db8-f20c0f1a6206"). InnerVolumeSpecName "kube-api-access-httvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:30.111667 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.111631 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:12:30.111667 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.111662 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:12:30.111667 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.111671 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-httvn\" (UniqueName: \"kubernetes.io/projected/79be7f9d-f615-4c9f-9db8-f20c0f1a6206-kube-api-access-httvn\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:12:30.187082 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.185600 5055 generic.go:356] "Generic (PLEG): container finished" podID="79be7f9d-f615-4c9f-9db8-f20c0f1a6206" containerID="5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667" exitCode=0 Oct 30 02:12:30.187082 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.185632 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" event={"ID":"79be7f9d-f615-4c9f-9db8-f20c0f1a6206","Type":"ContainerDied","Data":"5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667"} Oct 30 02:12:30.187082 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.185669 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" Oct 30 02:12:30.187082 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.185692 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h" event={"ID":"79be7f9d-f615-4c9f-9db8-f20c0f1a6206","Type":"ContainerDied","Data":"749d9205b1fc03d053567eb502804048c04f9951e76621173bd12416d5278290"} Oct 30 02:12:30.187082 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.185716 5055 scope.go:122] "RemoveContainer" containerID="5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667" Oct 30 02:12:30.202374 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.202351 5055 scope.go:122] "RemoveContainer" containerID="5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667" Oct 30 02:12:30.202693 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:12:30.202670 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667\": container with ID starting with 5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667 not found: ID does not exist" containerID="5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667" Oct 30 02:12:30.202796 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.202706 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667"} err="failed to get container status \"5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667\": rpc error: code = NotFound desc = could not find container \"5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667\": container with ID starting with 5d6150652341416db4f42090830d49ceb78c6b089ecc8c3de6f694ba6f097667 not found: ID does not exist" Oct 30 02:12:30.241534 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.241492 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h"] Oct 30 02:12:30.247475 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.247444 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-717065/test-model-717065-kserve-876d96bc9-x8v6h"] Oct 30 02:12:30.822875 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:30.822845 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79be7f9d-f615-4c9f-9db8-f20c0f1a6206" path="/var/lib/kubelet/pods/79be7f9d-f615-4c9f-9db8-f20c0f1a6206/volumes" Oct 30 02:12:31.155576 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:31.155504 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:12:49.289459 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:12:49.289412 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:12:49.857820 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:49.857783 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v"] Oct 30 02:12:49.858023 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:49.857967 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" podUID="49dc7504-c3b9-42da-bfcc-78a51040b9c2" containerName="sim" containerID="cri-o://10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0" gracePeriod=30 Oct 30 02:12:50.247701 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.247674 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:12:50.347289 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.345623 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49dc7504-c3b9-42da-bfcc-78a51040b9c2-tls\" (UniqueName: \"kubernetes.io/secret/49dc7504-c3b9-42da-bfcc-78a51040b9c2-tls\") pod \"49dc7504-c3b9-42da-bfcc-78a51040b9c2\" (UID: \"49dc7504-c3b9-42da-bfcc-78a51040b9c2\") " Oct 30 02:12:50.347289 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.345996 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49dc7504-c3b9-42da-bfcc-78a51040b9c2-kube-api-access-7dqpp\" (UniqueName: \"kubernetes.io/projected/49dc7504-c3b9-42da-bfcc-78a51040b9c2-kube-api-access-7dqpp\") pod \"49dc7504-c3b9-42da-bfcc-78a51040b9c2\" (UID: \"49dc7504-c3b9-42da-bfcc-78a51040b9c2\") " Oct 30 02:12:50.349788 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.349752 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49dc7504-c3b9-42da-bfcc-78a51040b9c2-tls" pod "49dc7504-c3b9-42da-bfcc-78a51040b9c2" (UID: "49dc7504-c3b9-42da-bfcc-78a51040b9c2"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:50.350642 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.350603 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49dc7504-c3b9-42da-bfcc-78a51040b9c2-kube-api-access-7dqpp" pod "49dc7504-c3b9-42da-bfcc-78a51040b9c2" (UID: "49dc7504-c3b9-42da-bfcc-78a51040b9c2"). InnerVolumeSpecName "kube-api-access-7dqpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:50.446381 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.445015 5055 generic.go:356] "Generic (PLEG): container finished" podID="49dc7504-c3b9-42da-bfcc-78a51040b9c2" containerID="10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0" exitCode=0 Oct 30 02:12:50.446381 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.445165 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" event={"ID":"49dc7504-c3b9-42da-bfcc-78a51040b9c2","Type":"ContainerDied","Data":"10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0"} Oct 30 02:12:50.446381 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.445202 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" event={"ID":"49dc7504-c3b9-42da-bfcc-78a51040b9c2","Type":"ContainerDied","Data":"fd23384c206404e43c08210c85e106a1465d343f4e9cfe2a96af791f99bfe0bc"} Oct 30 02:12:50.446381 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.445224 5055 scope.go:122] "RemoveContainer" containerID="10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0" Oct 30 02:12:50.446381 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.445401 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v" Oct 30 02:12:50.450872 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.450820 5055 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/49dc7504-c3b9-42da-bfcc-78a51040b9c2-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:12:50.450872 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.450849 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dqpp\" (UniqueName: \"kubernetes.io/projected/49dc7504-c3b9-42da-bfcc-78a51040b9c2-kube-api-access-7dqpp\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:12:50.528401 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.522994 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v"] Oct 30 02:12:50.541116 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.540698 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-extmod-shr-user-72f9d5/e2e-usage-sim-2bb30d-6b56449588-pj79v"] Oct 30 02:12:50.542566 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.542541 5055 scope.go:122] "RemoveContainer" containerID="10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0" Oct 30 02:12:50.543024 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:12:50.542995 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0\": container with ID starting with 10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0 not found: ID does not exist" containerID="10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0" Oct 30 02:12:50.543126 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.543037 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0"} err="failed to get container status \"10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0\": rpc error: code = NotFound desc = could not find container \"10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0\": container with ID starting with 10bfccc25c3071f58eb18834c1aa29dc6cc604e9953cd07fde5970d514c3e8c0 not found: ID does not exist" Oct 30 02:12:50.856846 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:12:50.856727 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49dc7504-c3b9-42da-bfcc-78a51040b9c2" path="/var/lib/kubelet/pods/49dc7504-c3b9-42da-bfcc-78a51040b9c2/volumes" Oct 30 02:13:19.222497 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:13:19.222454 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:13:38.140881 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:38.140614 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km"] Oct 30 02:13:38.142243 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:38.142212 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" containerID="cri-o://9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96" gracePeriod=30 Oct 30 02:13:43.463838 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.463809 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 02:13:43.596767 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.596726 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istiod-ca-cert\") pod \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " Oct 30 02:13:43.596977 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.596785 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-certs\") pod \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " Oct 30 02:13:43.596977 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.596851 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-socket\") pod \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " Oct 30 02:13:43.596977 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.596960 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-credential-socket\") pod \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " Oct 30 02:13:43.597142 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.597087 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-envoy\") pod \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " Oct 30 02:13:43.597199 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.597132 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-certs" pod "7458b5bd-13a6-4044-a5cb-66160c5dd31a" (UID: "7458b5bd-13a6-4044-a5cb-66160c5dd31a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:43.597256 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.597225 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-credential-socket" pod "7458b5bd-13a6-4044-a5cb-66160c5dd31a" (UID: "7458b5bd-13a6-4044-a5cb-66160c5dd31a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:43.597256 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.597225 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-socket" pod "7458b5bd-13a6-4044-a5cb-66160c5dd31a" (UID: "7458b5bd-13a6-4044-a5cb-66160c5dd31a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:43.597388 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.597264 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istiod-ca-cert" pod "7458b5bd-13a6-4044-a5cb-66160c5dd31a" (UID: "7458b5bd-13a6-4044-a5cb-66160c5dd31a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:43.597450 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.597420 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-data" pod "7458b5bd-13a6-4044-a5cb-66160c5dd31a" (UID: "7458b5bd-13a6-4044-a5cb-66160c5dd31a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:43.597575 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.597536 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-data\") pod \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " Oct 30 02:13:43.597672 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.597656 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-podinfo\") pod \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " Oct 30 02:13:43.597792 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.597697 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-token\" (UniqueName: \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-token\") pod \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " Oct 30 02:13:43.597792 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.597751 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-kube-api-access-tzj6s\" (UniqueName: \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-kube-api-access-tzj6s\") pod \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\" (UID: \"7458b5bd-13a6-4044-a5cb-66160c5dd31a\") " Oct 30 02:13:43.598498 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.598474 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:13:43.598743 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.598680 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:13:43.598865 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.598849 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:13:43.598951 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.598941 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:13:43.599028 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.599017 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:13:43.600287 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.600263 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-kube-api-access-tzj6s" pod "7458b5bd-13a6-4044-a5cb-66160c5dd31a" (UID: "7458b5bd-13a6-4044-a5cb-66160c5dd31a"). InnerVolumeSpecName "kube-api-access-tzj6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:43.600436 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.600286 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-podinfo" pod "7458b5bd-13a6-4044-a5cb-66160c5dd31a" (UID: "7458b5bd-13a6-4044-a5cb-66160c5dd31a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:13:43.600595 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.600571 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-envoy" pod "7458b5bd-13a6-4044-a5cb-66160c5dd31a" (UID: "7458b5bd-13a6-4044-a5cb-66160c5dd31a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:43.602024 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.602002 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-token" pod "7458b5bd-13a6-4044-a5cb-66160c5dd31a" (UID: "7458b5bd-13a6-4044-a5cb-66160c5dd31a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:43.701073 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.700978 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:13:43.701073 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.701010 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:13:43.701073 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.701018 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:13:43.701073 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:43.701029 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzj6s\" (UniqueName: \"kubernetes.io/projected/7458b5bd-13a6-4044-a5cb-66160c5dd31a-kube-api-access-tzj6s\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:13:44.088307 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.088273 5055 generic.go:356] "Generic (PLEG): container finished" podID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerID="9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96" exitCode=0 Oct 30 02:13:44.088514 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.088338 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" Oct 30 02:13:44.088514 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.088358 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" event={"ID":"7458b5bd-13a6-4044-a5cb-66160c5dd31a","Type":"ContainerDied","Data":"9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96"} Oct 30 02:13:44.088514 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.088420 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km" event={"ID":"7458b5bd-13a6-4044-a5cb-66160c5dd31a","Type":"ContainerDied","Data":"c43b8348886d792928f1384bffb5077745fff17a205a525401e9250cf30bb282"} Oct 30 02:13:44.088514 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.088458 5055 scope.go:122] "RemoveContainer" containerID="9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96" Oct 30 02:13:44.101560 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.101521 5055 scope.go:122] "RemoveContainer" containerID="2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d" Oct 30 02:13:44.114775 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.114755 5055 scope.go:122] "RemoveContainer" containerID="9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96" Oct 30 02:13:44.115109 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:13:44.115087 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96\": container with ID starting with 9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96 not found: ID does not exist" containerID="9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96" Oct 30 02:13:44.115228 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.115116 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96"} err="failed to get container status \"9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96\": rpc error: code = NotFound desc = could not find container \"9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96\": container with ID starting with 9ae57e26376353145ebaf1cb4b03d1acf8996f8d1cc45266c64d4c787c26ab96 not found: ID does not exist" Oct 30 02:13:44.115228 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.115145 5055 scope.go:122] "RemoveContainer" containerID="2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d" Oct 30 02:13:44.115467 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:13:44.115448 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d\": container with ID starting with 2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d not found: ID does not exist" containerID="2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d" Oct 30 02:13:44.115534 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.115476 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d"} err="failed to get container status \"2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d\": rpc error: code = NotFound desc = could not find container \"2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d\": container with ID starting with 2d42acf936e95c63bde067eda0d21e95eb6923be037b72a08b3ceec3c40f276d not found: ID does not exist" Oct 30 02:13:44.154940 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.154909 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km"] Oct 30 02:13:44.167850 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.167818 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-817fd7-openshift-default-7ffc7d7dd5-hn4km"] Oct 30 02:13:44.821463 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:13:44.821435 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" path="/var/lib/kubelet/pods/7458b5bd-13a6-4044-a5cb-66160c5dd31a/volumes" Oct 30 02:13:49.229433 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:13:49.229395 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:14:01.040918 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.040878 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7"] Oct 30 02:14:01.041461 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.041078 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" podUID="902d35cf-e127-4756-9a4b-ac9225465e17" containerName="maas-api" containerID="cri-o://61a78128eb479988e0ccbf0ade276082e26f1d833afa26db453d42424c939a02" gracePeriod=30 Oct 30 02:14:01.229616 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.229574 5055 generic.go:356] "Generic (PLEG): container finished" podID="902d35cf-e127-4756-9a4b-ac9225465e17" containerID="61a78128eb479988e0ccbf0ade276082e26f1d833afa26db453d42424c939a02" exitCode=0 Oct 30 02:14:01.229801 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.229669 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" event={"ID":"902d35cf-e127-4756-9a4b-ac9225465e17","Type":"ContainerDied","Data":"61a78128eb479988e0ccbf0ade276082e26f1d833afa26db453d42424c939a02"} Oct 30 02:14:01.559674 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.559160 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:14:01.643982 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.643943 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-maas-api-tls\") pod \"902d35cf-e127-4756-9a4b-ac9225465e17\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " Oct 30 02:14:01.644158 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.644001 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/902d35cf-e127-4756-9a4b-ac9225465e17-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/902d35cf-e127-4756-9a4b-ac9225465e17-openshift-service-ca\") pod \"902d35cf-e127-4756-9a4b-ac9225465e17\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " Oct 30 02:14:01.644242 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.644184 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-metrics-certs\" (UniqueName: \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-metrics-certs\") pod \"902d35cf-e127-4756-9a4b-ac9225465e17\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " Oct 30 02:14:01.644242 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.644225 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/902d35cf-e127-4756-9a4b-ac9225465e17-kube-api-access-k92m7\" (UniqueName: \"kubernetes.io/projected/902d35cf-e127-4756-9a4b-ac9225465e17-kube-api-access-k92m7\") pod \"902d35cf-e127-4756-9a4b-ac9225465e17\" (UID: \"902d35cf-e127-4756-9a4b-ac9225465e17\") " Oct 30 02:14:01.646063 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.645619 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/902d35cf-e127-4756-9a4b-ac9225465e17-openshift-service-ca" pod "902d35cf-e127-4756-9a4b-ac9225465e17" (UID: "902d35cf-e127-4756-9a4b-ac9225465e17"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:01.648885 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.648858 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-maas-api-tls" pod "902d35cf-e127-4756-9a4b-ac9225465e17" (UID: "902d35cf-e127-4756-9a4b-ac9225465e17"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:14:01.650537 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.650500 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-metrics-certs" pod "902d35cf-e127-4756-9a4b-ac9225465e17" (UID: "902d35cf-e127-4756-9a4b-ac9225465e17"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:14:01.662386 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.662212 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/902d35cf-e127-4756-9a4b-ac9225465e17-kube-api-access-k92m7" pod "902d35cf-e127-4756-9a4b-ac9225465e17" (UID: "902d35cf-e127-4756-9a4b-ac9225465e17"). InnerVolumeSpecName "kube-api-access-k92m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:01.746264 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.746226 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:14:01.746264 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.746262 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/902d35cf-e127-4756-9a4b-ac9225465e17-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:14:01.746533 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.746277 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/902d35cf-e127-4756-9a4b-ac9225465e17-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:14:01.746533 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:01.746290 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k92m7\" (UniqueName: \"kubernetes.io/projected/902d35cf-e127-4756-9a4b-ac9225465e17-kube-api-access-k92m7\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:14:02.238110 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:02.238081 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" Oct 30 02:14:02.238110 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:02.238104 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7" event={"ID":"902d35cf-e127-4756-9a4b-ac9225465e17","Type":"ContainerDied","Data":"554af6e375f4a1b5983a23e74778dd4e176bfe564f57ffb8c608eeaa0471718d"} Oct 30 02:14:02.238748 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:02.238175 5055 scope.go:122] "RemoveContainer" containerID="61a78128eb479988e0ccbf0ade276082e26f1d833afa26db453d42424c939a02" Oct 30 02:14:02.291855 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:02.291822 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7"] Oct 30 02:14:02.295640 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:02.295614 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f81908-86d586699-mrqg7"] Oct 30 02:14:02.827035 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:02.826999 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="902d35cf-e127-4756-9a4b-ac9225465e17" path="/var/lib/kubelet/pods/902d35cf-e127-4756-9a4b-ac9225465e17/volumes" Oct 30 02:14:19.220501 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:14:19.220467 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:14:27.504299 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:27.504258 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt"] Oct 30 02:14:27.504936 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:27.504589 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" podUID="7c10d615-5cde-47ae-821b-7e37711e71cd" containerName="maas-api" containerID="cri-o://b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61" gracePeriod=30 Oct 30 02:14:28.144895 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.144863 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 02:14:28.219913 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.219876 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-metrics-certs\") pod \"7c10d615-5cde-47ae-821b-7e37711e71cd\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " Oct 30 02:14:28.220120 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.220010 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-maas-api-tls\") pod \"7c10d615-5cde-47ae-821b-7e37711e71cd\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " Oct 30 02:14:28.220206 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.220166 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c10d615-5cde-47ae-821b-7e37711e71cd-kube-api-access-26rws\" (UniqueName: \"kubernetes.io/projected/7c10d615-5cde-47ae-821b-7e37711e71cd-kube-api-access-26rws\") pod \"7c10d615-5cde-47ae-821b-7e37711e71cd\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " Oct 30 02:14:28.220400 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.220374 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c10d615-5cde-47ae-821b-7e37711e71cd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c10d615-5cde-47ae-821b-7e37711e71cd-openshift-service-ca\") pod \"7c10d615-5cde-47ae-821b-7e37711e71cd\" (UID: \"7c10d615-5cde-47ae-821b-7e37711e71cd\") " Oct 30 02:14:28.220809 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.220745 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c10d615-5cde-47ae-821b-7e37711e71cd-openshift-service-ca" pod "7c10d615-5cde-47ae-821b-7e37711e71cd" (UID: "7c10d615-5cde-47ae-821b-7e37711e71cd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:28.221113 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.221070 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c10d615-5cde-47ae-821b-7e37711e71cd-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:14:28.223128 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.223071 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-metrics-certs" pod "7c10d615-5cde-47ae-821b-7e37711e71cd" (UID: "7c10d615-5cde-47ae-821b-7e37711e71cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:14:28.223500 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.223473 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c10d615-5cde-47ae-821b-7e37711e71cd-kube-api-access-26rws" pod "7c10d615-5cde-47ae-821b-7e37711e71cd" (UID: "7c10d615-5cde-47ae-821b-7e37711e71cd"). InnerVolumeSpecName "kube-api-access-26rws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:28.224185 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.224162 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-maas-api-tls" pod "7c10d615-5cde-47ae-821b-7e37711e71cd" (UID: "7c10d615-5cde-47ae-821b-7e37711e71cd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:14:28.323026 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.322988 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:14:28.323026 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.323021 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c10d615-5cde-47ae-821b-7e37711e71cd-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:14:28.323026 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.323031 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26rws\" (UniqueName: \"kubernetes.io/projected/7c10d615-5cde-47ae-821b-7e37711e71cd-kube-api-access-26rws\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:14:28.488295 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.488256 5055 generic.go:356] "Generic (PLEG): container finished" podID="7c10d615-5cde-47ae-821b-7e37711e71cd" containerID="b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61" exitCode=0 Oct 30 02:14:28.488565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.488311 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" Oct 30 02:14:28.488565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.488362 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" event={"ID":"7c10d615-5cde-47ae-821b-7e37711e71cd","Type":"ContainerDied","Data":"b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61"} Oct 30 02:14:28.488565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.488428 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt" event={"ID":"7c10d615-5cde-47ae-821b-7e37711e71cd","Type":"ContainerDied","Data":"bc7ba623b275234bbe23f867eee581592feba7aa697090b0a0c1206d8d61967c"} Oct 30 02:14:28.488565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.488462 5055 scope.go:122] "RemoveContainer" containerID="b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61" Oct 30 02:14:28.510820 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.510648 5055 scope.go:122] "RemoveContainer" containerID="b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61" Oct 30 02:14:28.511140 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:14:28.511059 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61\": container with ID starting with b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61 not found: ID does not exist" containerID="b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61" Oct 30 02:14:28.511140 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.511101 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61"} err="failed to get container status \"b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61\": rpc error: code = NotFound desc = could not find container \"b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61\": container with ID starting with b8a31f3e9eb32803a706eefd5b453486fa6f8400c184261f40402aea59fe9c61 not found: ID does not exist" Oct 30 02:14:28.546268 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.546232 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt"] Oct 30 02:14:28.550272 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.550238 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f6e11e-5bc5cdbb9f-fc5zt"] Oct 30 02:14:28.822177 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:14:28.822101 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c10d615-5cde-47ae-821b-7e37711e71cd" path="/var/lib/kubelet/pods/7c10d615-5cde-47ae-821b-7e37711e71cd/volumes" Oct 30 02:14:49.225040 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:14:49.225005 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:15:19.237695 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:15:19.237658 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:15:20.986906 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:20.986868 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v"] Oct 30 02:15:20.987579 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:20.987082 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" podUID="a7381e33-dd3e-4243-818d-21413e7e163e" containerName="maas-api" containerID="cri-o://62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac" gracePeriod=30 Oct 30 02:15:21.644379 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.642763 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 02:15:21.691571 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.691526 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7381e33-dd3e-4243-818d-21413e7e163e-kube-api-access-kpvsx\" (UniqueName: \"kubernetes.io/projected/a7381e33-dd3e-4243-818d-21413e7e163e-kube-api-access-kpvsx\") pod \"a7381e33-dd3e-4243-818d-21413e7e163e\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " Oct 30 02:15:21.691771 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.691593 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7381e33-dd3e-4243-818d-21413e7e163e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7381e33-dd3e-4243-818d-21413e7e163e-openshift-service-ca\") pod \"a7381e33-dd3e-4243-818d-21413e7e163e\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " Oct 30 02:15:21.691771 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.691656 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-maas-api-tls\") pod \"a7381e33-dd3e-4243-818d-21413e7e163e\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " Oct 30 02:15:21.691876 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.691770 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-metrics-certs\") pod \"a7381e33-dd3e-4243-818d-21413e7e163e\" (UID: \"a7381e33-dd3e-4243-818d-21413e7e163e\") " Oct 30 02:15:21.695442 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.693221 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7381e33-dd3e-4243-818d-21413e7e163e-openshift-service-ca" pod "a7381e33-dd3e-4243-818d-21413e7e163e" (UID: "a7381e33-dd3e-4243-818d-21413e7e163e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:21.695442 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.693993 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7381e33-dd3e-4243-818d-21413e7e163e-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.699346 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.697549 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-maas-api-tls" pod "a7381e33-dd3e-4243-818d-21413e7e163e" (UID: "a7381e33-dd3e-4243-818d-21413e7e163e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:21.716451 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.715860 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-metrics-certs" pod "a7381e33-dd3e-4243-818d-21413e7e163e" (UID: "a7381e33-dd3e-4243-818d-21413e7e163e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:21.716451 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.716061 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7381e33-dd3e-4243-818d-21413e7e163e-kube-api-access-kpvsx" pod "a7381e33-dd3e-4243-818d-21413e7e163e" (UID: "a7381e33-dd3e-4243-818d-21413e7e163e"). InnerVolumeSpecName "kube-api-access-kpvsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:21.795832 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.795795 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpvsx\" (UniqueName: \"kubernetes.io/projected/a7381e33-dd3e-4243-818d-21413e7e163e-kube-api-access-kpvsx\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.795832 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.795833 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.796175 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.795847 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7381e33-dd3e-4243-818d-21413e7e163e-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:15:21.983366 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.983288 5055 generic.go:356] "Generic (PLEG): container finished" podID="a7381e33-dd3e-4243-818d-21413e7e163e" containerID="62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac" exitCode=0 Oct 30 02:15:21.983565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.983399 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" Oct 30 02:15:21.983565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.983418 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" event={"ID":"a7381e33-dd3e-4243-818d-21413e7e163e","Type":"ContainerDied","Data":"62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac"} Oct 30 02:15:21.983565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.983488 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v" event={"ID":"a7381e33-dd3e-4243-818d-21413e7e163e","Type":"ContainerDied","Data":"c885e54feb658ba73088d9e434df1cf21b4d6a2b0944b1afe8528070f7dd4674"} Oct 30 02:15:21.983565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.983519 5055 scope.go:122] "RemoveContainer" containerID="62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac" Oct 30 02:15:21.998501 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.998277 5055 scope.go:122] "RemoveContainer" containerID="62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac" Oct 30 02:15:21.998819 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:15:21.998681 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac\": container with ID starting with 62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac not found: ID does not exist" containerID="62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac" Oct 30 02:15:21.998819 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:21.998725 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac"} err="failed to get container status \"62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac\": rpc error: code = NotFound desc = could not find container \"62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac\": container with ID starting with 62022d5841f9a972a573313cda4c2eac1fc15bd63f9a162864303b8be9829dac not found: ID does not exist" Oct 30 02:15:22.046255 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:22.046217 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v"] Oct 30 02:15:22.053490 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:22.053460 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-292018-7d8b96774b-zn52v"] Oct 30 02:15:22.825225 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:22.825188 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7381e33-dd3e-4243-818d-21413e7e163e" path="/var/lib/kubelet/pods/a7381e33-dd3e-4243-818d-21413e7e163e/volumes" Oct 30 02:15:49.221726 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:15:49.221677 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:15:55.155757 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:55.155716 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc"] Oct 30 02:15:59.676202 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:59.676163 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz"] Oct 30 02:15:59.677137 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:15:59.677102 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" containerID="cri-o://d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932" gracePeriod=30 Oct 30 02:16:04.986212 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:04.986177 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 02:16:05.105835 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.105799 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-certs\") pod \"fc69473a-f21a-4e8a-83da-2f2469a423bd\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " Oct 30 02:16:05.106033 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.105865 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc69473a-f21a-4e8a-83da-2f2469a423bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc69473a-f21a-4e8a-83da-2f2469a423bd-istiod-ca-cert\") pod \"fc69473a-f21a-4e8a-83da-2f2469a423bd\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " Oct 30 02:16:05.106033 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.105912 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-token\" (UniqueName: \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-token\") pod \"fc69473a-f21a-4e8a-83da-2f2469a423bd\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " Oct 30 02:16:05.106033 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.105972 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-envoy\") pod \"fc69473a-f21a-4e8a-83da-2f2469a423bd\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " Oct 30 02:16:05.106033 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.106005 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-credential-socket\") pod \"fc69473a-f21a-4e8a-83da-2f2469a423bd\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " Oct 30 02:16:05.106380 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.106087 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-podinfo\") pod \"fc69473a-f21a-4e8a-83da-2f2469a423bd\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " Oct 30 02:16:05.106380 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.106143 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-certs" pod "fc69473a-f21a-4e8a-83da-2f2469a423bd" (UID: "fc69473a-f21a-4e8a-83da-2f2469a423bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:05.106380 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.106247 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-socket\") pod \"fc69473a-f21a-4e8a-83da-2f2469a423bd\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " Oct 30 02:16:05.106562 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.106400 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-credential-socket" pod "fc69473a-f21a-4e8a-83da-2f2469a423bd" (UID: "fc69473a-f21a-4e8a-83da-2f2469a423bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:05.106562 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.106442 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-data\") pod \"fc69473a-f21a-4e8a-83da-2f2469a423bd\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " Oct 30 02:16:05.106562 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.106517 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-socket" pod "fc69473a-f21a-4e8a-83da-2f2469a423bd" (UID: "fc69473a-f21a-4e8a-83da-2f2469a423bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:05.106562 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.106553 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-kube-api-access-sgvpd\" (UniqueName: \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-kube-api-access-sgvpd\") pod \"fc69473a-f21a-4e8a-83da-2f2469a423bd\" (UID: \"fc69473a-f21a-4e8a-83da-2f2469a423bd\") " Oct 30 02:16:05.106831 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.106546 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc69473a-f21a-4e8a-83da-2f2469a423bd-istiod-ca-cert" pod "fc69473a-f21a-4e8a-83da-2f2469a423bd" (UID: "fc69473a-f21a-4e8a-83da-2f2469a423bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:05.107012 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.106735 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-data" pod "fc69473a-f21a-4e8a-83da-2f2469a423bd" (UID: "fc69473a-f21a-4e8a-83da-2f2469a423bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:05.107541 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.107514 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.107650 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.107545 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.107650 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.107561 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.107650 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.107574 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc69473a-f21a-4e8a-83da-2f2469a423bd-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.107650 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.107588 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.109021 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.108992 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-envoy" pod "fc69473a-f21a-4e8a-83da-2f2469a423bd" (UID: "fc69473a-f21a-4e8a-83da-2f2469a423bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:05.109146 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.109023 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-kube-api-access-sgvpd" pod "fc69473a-f21a-4e8a-83da-2f2469a423bd" (UID: "fc69473a-f21a-4e8a-83da-2f2469a423bd"). InnerVolumeSpecName "kube-api-access-sgvpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:05.109265 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.109239 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-token" pod "fc69473a-f21a-4e8a-83da-2f2469a423bd" (UID: "fc69473a-f21a-4e8a-83da-2f2469a423bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:05.109511 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.109476 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-podinfo" pod "fc69473a-f21a-4e8a-83da-2f2469a423bd" (UID: "fc69473a-f21a-4e8a-83da-2f2469a423bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:16:05.210406 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.210294 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.210406 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.210348 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgvpd\" (UniqueName: \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-kube-api-access-sgvpd\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.210406 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.210360 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.210406 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.210368 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc69473a-f21a-4e8a-83da-2f2469a423bd-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:05.388622 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.388580 5055 generic.go:356] "Generic (PLEG): container finished" podID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerID="d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932" exitCode=0 Oct 30 02:16:05.388830 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.388634 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" Oct 30 02:16:05.388830 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.388663 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" event={"ID":"fc69473a-f21a-4e8a-83da-2f2469a423bd","Type":"ContainerDied","Data":"d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932"} Oct 30 02:16:05.388830 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.388726 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz" event={"ID":"fc69473a-f21a-4e8a-83da-2f2469a423bd","Type":"ContainerDied","Data":"eb23b5b9d9667c3f77e1e8e7cd215c14e09b09592e9b53f03634484064af2c01"} Oct 30 02:16:05.388830 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.388754 5055 scope.go:122] "RemoveContainer" containerID="d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932" Oct 30 02:16:05.404812 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.404782 5055 scope.go:122] "RemoveContainer" containerID="bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85" Oct 30 02:16:05.416894 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.416872 5055 scope.go:122] "RemoveContainer" containerID="d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932" Oct 30 02:16:05.417220 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:16:05.417193 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932\": container with ID starting with d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932 not found: ID does not exist" containerID="d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932" Oct 30 02:16:05.417315 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.417232 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932"} err="failed to get container status \"d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932\": rpc error: code = NotFound desc = could not find container \"d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932\": container with ID starting with d715f2a519cf8a35f6823defe164ea08c8779369370a4aadb72493a9a7f84932 not found: ID does not exist" Oct 30 02:16:05.417315 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.417257 5055 scope.go:122] "RemoveContainer" containerID="bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85" Oct 30 02:16:05.417602 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:16:05.417581 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85\": container with ID starting with bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85 not found: ID does not exist" containerID="bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85" Oct 30 02:16:05.417688 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.417611 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85"} err="failed to get container status \"bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85\": rpc error: code = NotFound desc = could not find container \"bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85\": container with ID starting with bb36757acea77da015b7ed2ec9d73a26cc82c979557f7671a60abbc383935d85 not found: ID does not exist" Oct 30 02:16:05.445111 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.445079 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz"] Oct 30 02:16:05.465759 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:05.465673 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-cd8173-openshift-default-86d54bff69-5xrlz"] Oct 30 02:16:06.822251 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:06.822222 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" path="/var/lib/kubelet/pods/fc69473a-f21a-4e8a-83da-2f2469a423bd/volumes" Oct 30 02:16:10.181985 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.181936 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" podUID="baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" containerName="main" containerID="cri-o://b9ad353838f6992950c557beead1153c9e7b679e400af7797e48e58b9e78cb1c" gracePeriod=45 Oct 30 02:16:10.434859 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.434787 5055 generic.go:356] "Generic (PLEG): container finished" podID="baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" containerID="b9ad353838f6992950c557beead1153c9e7b679e400af7797e48e58b9e78cb1c" exitCode=0 Oct 30 02:16:10.435003 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.434855 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" event={"ID":"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c","Type":"ContainerDied","Data":"b9ad353838f6992950c557beead1153c9e7b679e400af7797e48e58b9e78cb1c"} Oct 30 02:16:10.497972 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.497943 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:16:10.592540 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.592504 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tmp-dir\") pod \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " Oct 30 02:16:10.592754 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.592586 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-home\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-home\") pod \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " Oct 30 02:16:10.592754 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.592656 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-kube-api-access-pbjz9\" (UniqueName: \"kubernetes.io/projected/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-kube-api-access-pbjz9\") pod \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " Oct 30 02:16:10.592754 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.592689 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-model-cache\") pod \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " Oct 30 02:16:10.592948 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.592824 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tmp-dir" pod "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" (UID: "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:10.592948 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.592870 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-home" pod "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" (UID: "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:10.593054 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.592968 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-dshm\") pod \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " Oct 30 02:16:10.593121 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.593099 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-model-cache" pod "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" (UID: "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:10.593185 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.593127 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tls-certs\") pod \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\" (UID: \"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c\") " Oct 30 02:16:10.594472 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.594445 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:10.594472 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.594474 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:10.594781 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.594487 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:10.595529 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.595506 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-dshm" pod "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" (UID: "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:10.595680 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.595627 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-kube-api-access-pbjz9" pod "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" (UID: "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c"). InnerVolumeSpecName "kube-api-access-pbjz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:10.595890 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.595873 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tls-certs" pod "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" (UID: "baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:10.696539 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.696443 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:10.696539 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.696479 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:10.696539 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:10.696494 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbjz9\" (UniqueName: \"kubernetes.io/projected/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c-kube-api-access-pbjz9\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:11.164961 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.164925 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p"] Oct 30 02:16:11.165185 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.165160 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" podUID="fee10591-c2b7-4290-a568-e6d39436988f" containerName="maas-api" containerID="cri-o://d0efa95b5ba90944a45f532ac2b06ab0ba3a2594a331113db174103a7a746707" gracePeriod=30 Oct 30 02:16:11.445420 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.444422 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" podUID="baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.192:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:16:11.579351 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.573634 5055 generic.go:356] "Generic (PLEG): container finished" podID="fee10591-c2b7-4290-a568-e6d39436988f" containerID="d0efa95b5ba90944a45f532ac2b06ab0ba3a2594a331113db174103a7a746707" exitCode=0 Oct 30 02:16:11.579351 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.573870 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" event={"ID":"fee10591-c2b7-4290-a568-e6d39436988f","Type":"ContainerDied","Data":"d0efa95b5ba90944a45f532ac2b06ab0ba3a2594a331113db174103a7a746707"} Oct 30 02:16:11.583348 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.580542 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" event={"ID":"baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c","Type":"ContainerDied","Data":"310a146a1988322a648fdef414fb385393bdac0b354617ceb0b3823ddf121c04"} Oct 30 02:16:11.583348 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.580601 5055 scope.go:122] "RemoveContainer" containerID="b9ad353838f6992950c557beead1153c9e7b679e400af7797e48e58b9e78cb1c" Oct 30 02:16:11.583348 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.580762 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc" Oct 30 02:16:11.743431 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.743288 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 02:16:11.753505 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.753470 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc"] Oct 30 02:16:11.759005 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.758973 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-aaf045/test-model-aaf045-kserve-c55566bb9-jslrc"] Oct 30 02:16:11.873849 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.873807 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-maas-api-tls\") pod \"fee10591-c2b7-4290-a568-e6d39436988f\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " Oct 30 02:16:11.874031 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.873900 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee10591-c2b7-4290-a568-e6d39436988f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee10591-c2b7-4290-a568-e6d39436988f-openshift-service-ca\") pod \"fee10591-c2b7-4290-a568-e6d39436988f\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " Oct 30 02:16:11.874031 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.873950 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee10591-c2b7-4290-a568-e6d39436988f-kube-api-access-cr6g7\" (UniqueName: \"kubernetes.io/projected/fee10591-c2b7-4290-a568-e6d39436988f-kube-api-access-cr6g7\") pod \"fee10591-c2b7-4290-a568-e6d39436988f\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " Oct 30 02:16:11.874031 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.873996 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-metrics-certs\") pod \"fee10591-c2b7-4290-a568-e6d39436988f\" (UID: \"fee10591-c2b7-4290-a568-e6d39436988f\") " Oct 30 02:16:11.874354 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.874300 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee10591-c2b7-4290-a568-e6d39436988f-openshift-service-ca" pod "fee10591-c2b7-4290-a568-e6d39436988f" (UID: "fee10591-c2b7-4290-a568-e6d39436988f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:11.874991 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.874959 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fee10591-c2b7-4290-a568-e6d39436988f-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:11.876610 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.876564 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee10591-c2b7-4290-a568-e6d39436988f-kube-api-access-cr6g7" pod "fee10591-c2b7-4290-a568-e6d39436988f" (UID: "fee10591-c2b7-4290-a568-e6d39436988f"). InnerVolumeSpecName "kube-api-access-cr6g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:11.876610 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.876589 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-metrics-certs" pod "fee10591-c2b7-4290-a568-e6d39436988f" (UID: "fee10591-c2b7-4290-a568-e6d39436988f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:11.876781 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.876679 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-maas-api-tls" pod "fee10591-c2b7-4290-a568-e6d39436988f" (UID: "fee10591-c2b7-4290-a568-e6d39436988f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:11.976634 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.976506 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:11.976634 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.976544 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr6g7\" (UniqueName: \"kubernetes.io/projected/fee10591-c2b7-4290-a568-e6d39436988f-kube-api-access-cr6g7\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:11.976634 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:11.976561 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fee10591-c2b7-4290-a568-e6d39436988f-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:12.592841 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:12.592799 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" Oct 30 02:16:12.593321 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:12.592802 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p" event={"ID":"fee10591-c2b7-4290-a568-e6d39436988f","Type":"ContainerDied","Data":"45d492e764243467efa83278d3ce06bcdbaf57eb0af014772ec61a806a500d3c"} Oct 30 02:16:12.593321 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:12.592963 5055 scope.go:122] "RemoveContainer" containerID="d0efa95b5ba90944a45f532ac2b06ab0ba3a2594a331113db174103a7a746707" Oct 30 02:16:12.648436 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:12.648403 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p"] Oct 30 02:16:12.658733 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:12.658697 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-b6b7fc-d54bc6f86-ngg5p"] Oct 30 02:16:12.822430 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:12.822398 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" path="/var/lib/kubelet/pods/baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c/volumes" Oct 30 02:16:12.822883 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:12.822869 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee10591-c2b7-4290-a568-e6d39436988f" path="/var/lib/kubelet/pods/fee10591-c2b7-4290-a568-e6d39436988f/volumes" Oct 30 02:16:19.234965 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:16:19.234923 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:16:32.061528 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:32.061482 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8"] Oct 30 02:16:32.074564 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:32.074534 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps"] Oct 30 02:16:45.014926 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.014705 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk"] Oct 30 02:16:45.015446 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.014968 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" podUID="43fbb343-2521-4767-b229-0912ab4880d1" containerName="maas-api" containerID="cri-o://ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765" gracePeriod=30 Oct 30 02:16:45.580862 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.580813 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:16:45.607875 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.607839 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-maas-api-tls\") pod \"43fbb343-2521-4767-b229-0912ab4880d1\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " Oct 30 02:16:45.608074 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.607993 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fbb343-2521-4767-b229-0912ab4880d1-kube-api-access-frh6b\" (UniqueName: \"kubernetes.io/projected/43fbb343-2521-4767-b229-0912ab4880d1-kube-api-access-frh6b\") pod \"43fbb343-2521-4767-b229-0912ab4880d1\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " Oct 30 02:16:45.608074 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.608038 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43fbb343-2521-4767-b229-0912ab4880d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43fbb343-2521-4767-b229-0912ab4880d1-openshift-service-ca\") pod \"43fbb343-2521-4767-b229-0912ab4880d1\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " Oct 30 02:16:45.608262 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.608240 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-metrics-certs\") pod \"43fbb343-2521-4767-b229-0912ab4880d1\" (UID: \"43fbb343-2521-4767-b229-0912ab4880d1\") " Oct 30 02:16:45.608546 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.608519 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fbb343-2521-4767-b229-0912ab4880d1-openshift-service-ca" pod "43fbb343-2521-4767-b229-0912ab4880d1" (UID: "43fbb343-2521-4767-b229-0912ab4880d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:45.608946 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.608919 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/43fbb343-2521-4767-b229-0912ab4880d1-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:45.610493 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.610468 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-metrics-certs" pod "43fbb343-2521-4767-b229-0912ab4880d1" (UID: "43fbb343-2521-4767-b229-0912ab4880d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:45.610721 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.610699 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-maas-api-tls" pod "43fbb343-2521-4767-b229-0912ab4880d1" (UID: "43fbb343-2521-4767-b229-0912ab4880d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:45.610790 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.610759 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fbb343-2521-4767-b229-0912ab4880d1-kube-api-access-frh6b" pod "43fbb343-2521-4767-b229-0912ab4880d1" (UID: "43fbb343-2521-4767-b229-0912ab4880d1"). InnerVolumeSpecName "kube-api-access-frh6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:45.711532 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.711489 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:45.711532 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.711527 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/43fbb343-2521-4767-b229-0912ab4880d1-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:45.711734 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.711542 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frh6b\" (UniqueName: \"kubernetes.io/projected/43fbb343-2521-4767-b229-0912ab4880d1-kube-api-access-frh6b\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:45.854006 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.853919 5055 generic.go:356] "Generic (PLEG): container finished" podID="43fbb343-2521-4767-b229-0912ab4880d1" containerID="ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765" exitCode=0 Oct 30 02:16:45.854006 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.853975 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" Oct 30 02:16:45.854224 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.854001 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" event={"ID":"43fbb343-2521-4767-b229-0912ab4880d1","Type":"ContainerDied","Data":"ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765"} Oct 30 02:16:45.854224 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.854069 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk" event={"ID":"43fbb343-2521-4767-b229-0912ab4880d1","Type":"ContainerDied","Data":"956e242c6b7d4039d8bb940fe49b6697e21d01768184459a46aed0c632ba3f88"} Oct 30 02:16:45.854224 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.854099 5055 scope.go:122] "RemoveContainer" containerID="ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765" Oct 30 02:16:45.867564 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.867541 5055 scope.go:122] "RemoveContainer" containerID="ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765" Oct 30 02:16:45.867899 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:16:45.867878 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765\": container with ID starting with ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765 not found: ID does not exist" containerID="ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765" Oct 30 02:16:45.867967 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.867909 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765"} err="failed to get container status \"ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765\": rpc error: code = NotFound desc = could not find container \"ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765\": container with ID starting with ba8a1fa3b6e38381b18632dbe2fa6eda7c0f188321ddfeeaa2a5032649730765 not found: ID does not exist" Oct 30 02:16:45.932385 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.932347 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk"] Oct 30 02:16:45.946126 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:45.946095 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-aaf045-8589d7b975-695bk"] Oct 30 02:16:46.821914 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:46.821883 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43fbb343-2521-4767-b229-0912ab4880d1" path="/var/lib/kubelet/pods/43fbb343-2521-4767-b229-0912ab4880d1/volumes" Oct 30 02:16:47.085563 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.085458 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" containerID="cri-o://70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5" gracePeriod=45 Oct 30 02:16:47.098520 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.098475 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" containerID="cri-o://5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6" gracePeriod=45 Oct 30 02:16:47.198733 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.198698 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": dial tcp 10.132.0.76:8000: connect: connection refused" Oct 30 02:16:47.410444 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.407771 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 02:16:47.430919 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.430893 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 02:16:47.452837 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.452813 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-tmp-dir\") pod \"02cf2cd3-e5cd-4f83-968a-2694debfe092\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " Oct 30 02:16:47.452977 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.452858 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02cf2cd3-e5cd-4f83-968a-2694debfe092-tls-certs\" (UniqueName: \"kubernetes.io/secret/02cf2cd3-e5cd-4f83-968a-2694debfe092-tls-certs\") pod \"02cf2cd3-e5cd-4f83-968a-2694debfe092\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " Oct 30 02:16:47.453048 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.453030 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-model-cache\") pod \"02cf2cd3-e5cd-4f83-968a-2694debfe092\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " Oct 30 02:16:47.453123 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.453081 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02cf2cd3-e5cd-4f83-968a-2694debfe092-kube-api-access-4zb8f\" (UniqueName: \"kubernetes.io/projected/02cf2cd3-e5cd-4f83-968a-2694debfe092-kube-api-access-4zb8f\") pod \"02cf2cd3-e5cd-4f83-968a-2694debfe092\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " Oct 30 02:16:47.453123 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.453094 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-tmp-dir" pod "02cf2cd3-e5cd-4f83-968a-2694debfe092" (UID: "02cf2cd3-e5cd-4f83-968a-2694debfe092"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.453241 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.453145 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-dshm\") pod \"02cf2cd3-e5cd-4f83-968a-2694debfe092\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " Oct 30 02:16:47.453241 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.453194 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-home\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-home\") pod \"02cf2cd3-e5cd-4f83-968a-2694debfe092\" (UID: \"02cf2cd3-e5cd-4f83-968a-2694debfe092\") " Oct 30 02:16:47.453394 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.453297 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-model-cache" pod "02cf2cd3-e5cd-4f83-968a-2694debfe092" (UID: "02cf2cd3-e5cd-4f83-968a-2694debfe092"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.453705 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.453681 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-home" pod "02cf2cd3-e5cd-4f83-968a-2694debfe092" (UID: "02cf2cd3-e5cd-4f83-968a-2694debfe092"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.453951 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.453919 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.453951 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.453939 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.453951 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.453951 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.456401 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.456373 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-dshm" pod "02cf2cd3-e5cd-4f83-968a-2694debfe092" (UID: "02cf2cd3-e5cd-4f83-968a-2694debfe092"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.456542 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.456518 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02cf2cd3-e5cd-4f83-968a-2694debfe092-kube-api-access-4zb8f" pod "02cf2cd3-e5cd-4f83-968a-2694debfe092" (UID: "02cf2cd3-e5cd-4f83-968a-2694debfe092"). InnerVolumeSpecName "kube-api-access-4zb8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:47.457780 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.457755 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02cf2cd3-e5cd-4f83-968a-2694debfe092-tls-certs" pod "02cf2cd3-e5cd-4f83-968a-2694debfe092" (UID: "02cf2cd3-e5cd-4f83-968a-2694debfe092"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:47.555281 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.555241 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7888167-f48d-41cb-8a4a-dd9510bb9481-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7888167-f48d-41cb-8a4a-dd9510bb9481-tls-certs\") pod \"d7888167-f48d-41cb-8a4a-dd9510bb9481\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " Oct 30 02:16:47.555500 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.555313 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-dshm\") pod \"d7888167-f48d-41cb-8a4a-dd9510bb9481\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " Oct 30 02:16:47.555500 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.555433 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-tmp-dir\") pod \"d7888167-f48d-41cb-8a4a-dd9510bb9481\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " Oct 30 02:16:47.555750 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.555715 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-tmp-dir" pod "d7888167-f48d-41cb-8a4a-dd9510bb9481" (UID: "d7888167-f48d-41cb-8a4a-dd9510bb9481"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.555869 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.555848 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-model-cache\") pod \"d7888167-f48d-41cb-8a4a-dd9510bb9481\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " Oct 30 02:16:47.556052 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.556000 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-home\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-home\") pod \"d7888167-f48d-41cb-8a4a-dd9510bb9481\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " Oct 30 02:16:47.556141 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.556084 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-model-cache" pod "d7888167-f48d-41cb-8a4a-dd9510bb9481" (UID: "d7888167-f48d-41cb-8a4a-dd9510bb9481"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.556141 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.556127 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7888167-f48d-41cb-8a4a-dd9510bb9481-kube-api-access-gqsqm\" (UniqueName: \"kubernetes.io/projected/d7888167-f48d-41cb-8a4a-dd9510bb9481-kube-api-access-gqsqm\") pod \"d7888167-f48d-41cb-8a4a-dd9510bb9481\" (UID: \"d7888167-f48d-41cb-8a4a-dd9510bb9481\") " Oct 30 02:16:47.556350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.556299 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-home" pod "d7888167-f48d-41cb-8a4a-dd9510bb9481" (UID: "d7888167-f48d-41cb-8a4a-dd9510bb9481"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.557117 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.557090 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02cf2cd3-e5cd-4f83-968a-2694debfe092-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.557245 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.557150 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zb8f\" (UniqueName: \"kubernetes.io/projected/02cf2cd3-e5cd-4f83-968a-2694debfe092-kube-api-access-4zb8f\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.557245 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.557166 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.557245 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.557184 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02cf2cd3-e5cd-4f83-968a-2694debfe092-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.557245 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.557197 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.557245 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.557210 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.559407 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.559379 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-dshm" pod "d7888167-f48d-41cb-8a4a-dd9510bb9481" (UID: "d7888167-f48d-41cb-8a4a-dd9510bb9481"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:47.559558 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.559499 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7888167-f48d-41cb-8a4a-dd9510bb9481-tls-certs" pod "d7888167-f48d-41cb-8a4a-dd9510bb9481" (UID: "d7888167-f48d-41cb-8a4a-dd9510bb9481"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:47.560196 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.559715 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7888167-f48d-41cb-8a4a-dd9510bb9481-kube-api-access-gqsqm" pod "d7888167-f48d-41cb-8a4a-dd9510bb9481" (UID: "d7888167-f48d-41cb-8a4a-dd9510bb9481"). InnerVolumeSpecName "kube-api-access-gqsqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:47.658165 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.658087 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d7888167-f48d-41cb-8a4a-dd9510bb9481-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.658165 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.658117 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqsqm\" (UniqueName: \"kubernetes.io/projected/d7888167-f48d-41cb-8a4a-dd9510bb9481-kube-api-access-gqsqm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.658165 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.658128 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7888167-f48d-41cb-8a4a-dd9510bb9481-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:16:47.872521 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.872490 5055 generic.go:356] "Generic (PLEG): container finished" podID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerID="5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6" exitCode=0 Oct 30 02:16:47.872908 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.872629 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" event={"ID":"d7888167-f48d-41cb-8a4a-dd9510bb9481","Type":"ContainerDied","Data":"5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6"} Oct 30 02:16:47.872908 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.872660 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" event={"ID":"d7888167-f48d-41cb-8a4a-dd9510bb9481","Type":"ContainerDied","Data":"434099926a47cd9fc2b889e960a9410516e0055893c91a942859db0605b5cec5"} Oct 30 02:16:47.872908 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.872676 5055 scope.go:122] "RemoveContainer" containerID="5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6" Oct 30 02:16:47.872908 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.872751 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps" Oct 30 02:16:47.875582 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.875555 5055 generic.go:356] "Generic (PLEG): container finished" podID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerID="70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5" exitCode=0 Oct 30 02:16:47.875716 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.875623 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" Oct 30 02:16:47.875716 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.875680 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" event={"ID":"02cf2cd3-e5cd-4f83-968a-2694debfe092","Type":"ContainerDied","Data":"70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5"} Oct 30 02:16:47.875827 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.875715 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8" event={"ID":"02cf2cd3-e5cd-4f83-968a-2694debfe092","Type":"ContainerDied","Data":"c1c33f678d3ad4a3d359d513174b5a0243361ed3c2d6afa9113d5187fff49618"} Oct 30 02:16:47.886600 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.886579 5055 scope.go:122] "RemoveContainer" containerID="5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6" Oct 30 02:16:47.886875 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:16:47.886854 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6\": container with ID starting with 5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6 not found: ID does not exist" containerID="5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6" Oct 30 02:16:47.886939 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.886882 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6"} err="failed to get container status \"5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6\": rpc error: code = NotFound desc = could not find container \"5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6\": container with ID starting with 5781049941c4443efe3da5c87c2541d67836a64b9798f39005575c4a36a743b6 not found: ID does not exist" Oct 30 02:16:47.886939 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.886898 5055 scope.go:122] "RemoveContainer" containerID="70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5" Oct 30 02:16:47.898352 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.898315 5055 scope.go:122] "RemoveContainer" containerID="70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5" Oct 30 02:16:47.898682 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:16:47.898636 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5\": container with ID starting with 70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5 not found: ID does not exist" containerID="70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5" Oct 30 02:16:47.898682 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.898675 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5"} err="failed to get container status \"70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5\": rpc error: code = NotFound desc = could not find container \"70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5\": container with ID starting with 70d8b801a87305d44c75d164594bee77d8695352da57d90d733e14b7a41219b5 not found: ID does not exist" Oct 30 02:16:47.940481 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.940177 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8"] Oct 30 02:16:47.946012 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.945942 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-2-w4-803472-kserve-76646db648-xsqr8"] Oct 30 02:16:47.970961 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.970930 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps"] Oct 30 02:16:47.978284 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:47.978253 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-803472/distinct-w4-803472-kserve-8477c5f97f-cpsps"] Oct 30 02:16:48.821616 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:48.821587 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" path="/var/lib/kubelet/pods/02cf2cd3-e5cd-4f83-968a-2694debfe092/volumes" Oct 30 02:16:48.822150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:16:48.822135 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" path="/var/lib/kubelet/pods/d7888167-f48d-41cb-8a4a-dd9510bb9481/volumes" Oct 30 02:16:49.223137 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:16:49.223100 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:17:01.548568 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:01.548529 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm"] Oct 30 02:17:06.830856 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:06.830822 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw"] Oct 30 02:17:06.831278 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:06.831073 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" podUID="e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b" containerName="maas-api" containerID="cri-o://b933bf327dfe28ad1186dc0d797c44b095342cd22ae670650065e53094d288cb" gracePeriod=30 Oct 30 02:17:07.057164 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.057127 5055 generic.go:356] "Generic (PLEG): container finished" podID="e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b" containerID="b933bf327dfe28ad1186dc0d797c44b095342cd22ae670650065e53094d288cb" exitCode=0 Oct 30 02:17:07.057391 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.057206 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" event={"ID":"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b","Type":"ContainerDied","Data":"b933bf327dfe28ad1186dc0d797c44b095342cd22ae670650065e53094d288cb"} Oct 30 02:17:07.349046 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.346261 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 02:17:07.442740 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.442701 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-kube-api-access-pxpz6\" (UniqueName: \"kubernetes.io/projected/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-kube-api-access-pxpz6\") pod \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " Oct 30 02:17:07.442920 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.442830 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-maas-api-tls\") pod \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " Oct 30 02:17:07.442963 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.442930 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-openshift-service-ca\") pod \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " Oct 30 02:17:07.443000 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.442983 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-metrics-certs\") pod \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\" (UID: \"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b\") " Oct 30 02:17:07.443313 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.443284 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-openshift-service-ca" pod "e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b" (UID: "e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:07.443633 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.443615 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.445234 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.445211 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-kube-api-access-pxpz6" pod "e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b" (UID: "e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b"). InnerVolumeSpecName "kube-api-access-pxpz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:07.445609 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.445585 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-maas-api-tls" pod "e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b" (UID: "e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:07.446240 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.446216 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-metrics-certs" pod "e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b" (UID: "e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:07.545450 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.545367 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxpz6\" (UniqueName: \"kubernetes.io/projected/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-kube-api-access-pxpz6\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.545450 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.545396 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:07.545450 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:07.545404 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:08.084351 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:08.069233 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" event={"ID":"e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b","Type":"ContainerDied","Data":"f00c0b14b57431b3d1a0a93bdf59f35dd0cd8c41c7b38d052ad109017c56dc9f"} Oct 30 02:17:08.084351 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:08.069251 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw" Oct 30 02:17:08.084351 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:08.069369 5055 scope.go:122] "RemoveContainer" containerID="b933bf327dfe28ad1186dc0d797c44b095342cd22ae670650065e53094d288cb" Oct 30 02:17:08.122318 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:08.122282 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw"] Oct 30 02:17:08.134692 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:08.134661 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-264bdd-86544654b5-mjjgw"] Oct 30 02:17:08.824299 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:08.824269 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b" path="/var/lib/kubelet/pods/e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b/volumes" Oct 30 02:17:12.273067 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:12.273030 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5"] Oct 30 02:17:13.530806 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:13.530780 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:17:13.533234 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:13.533209 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:17:16.303046 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:16.303011 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5"] Oct 30 02:17:16.303602 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:16.303254 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" podUID="679f8047-1bb0-4123-92d7-6474c85ce2be" containerName="maas-api" containerID="cri-o://9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc" gracePeriod=30 Oct 30 02:17:16.576019 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:16.575922 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" podUID="28e91e96-2e4d-4df8-b9fa-928f401194c6" containerName="main" containerID="cri-o://552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1" gracePeriod=45 Oct 30 02:17:17.055765 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.055734 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:17:17.070527 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.070489 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:17:17.179248 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.179158 5055 generic.go:356] "Generic (PLEG): container finished" podID="28e91e96-2e4d-4df8-b9fa-928f401194c6" containerID="552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1" exitCode=0 Oct 30 02:17:17.179498 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.179250 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" event={"ID":"28e91e96-2e4d-4df8-b9fa-928f401194c6","Type":"ContainerDied","Data":"552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1"} Oct 30 02:17:17.179498 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.179287 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" event={"ID":"28e91e96-2e4d-4df8-b9fa-928f401194c6","Type":"ContainerDied","Data":"dafeaacd9f188fead37e0874e94373c638c1b8350596bcff1f9d1ad35fe4e825"} Oct 30 02:17:17.180208 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.180182 5055 scope.go:122] "RemoveContainer" containerID="552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1" Oct 30 02:17:17.180438 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.180417 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm" Oct 30 02:17:17.184009 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.183830 5055 generic.go:356] "Generic (PLEG): container finished" podID="679f8047-1bb0-4123-92d7-6474c85ce2be" containerID="9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc" exitCode=0 Oct 30 02:17:17.184168 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.184092 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" event={"ID":"679f8047-1bb0-4123-92d7-6474c85ce2be","Type":"ContainerDied","Data":"9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc"} Oct 30 02:17:17.184168 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.184159 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" event={"ID":"679f8047-1bb0-4123-92d7-6474c85ce2be","Type":"ContainerDied","Data":"30b7fb1b6efc984399271099c6b10b57a3d33a3c9ec5f7633a5afba2e5eefe30"} Oct 30 02:17:17.184956 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.184459 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" Oct 30 02:17:17.203441 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.203408 5055 scope.go:122] "RemoveContainer" containerID="552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1" Oct 30 02:17:17.207060 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:17:17.207024 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1\": container with ID starting with 552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1 not found: ID does not exist" containerID="552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1" Oct 30 02:17:17.207950 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.207923 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-tmp-dir\") pod \"28e91e96-2e4d-4df8-b9fa-928f401194c6\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " Oct 30 02:17:17.208058 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.207971 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-metrics-certs\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-metrics-certs\") pod \"679f8047-1bb0-4123-92d7-6474c85ce2be\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " Oct 30 02:17:17.208058 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.208019 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-dshm\") pod \"28e91e96-2e4d-4df8-b9fa-928f401194c6\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " Oct 30 02:17:17.208181 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.208089 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-home\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-home\") pod \"28e91e96-2e4d-4df8-b9fa-928f401194c6\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " Oct 30 02:17:17.208181 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.208119 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/679f8047-1bb0-4123-92d7-6474c85ce2be-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679f8047-1bb0-4123-92d7-6474c85ce2be-openshift-service-ca\") pod \"679f8047-1bb0-4123-92d7-6474c85ce2be\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " Oct 30 02:17:17.208181 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.208163 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e91e96-2e4d-4df8-b9fa-928f401194c6-kube-api-access-6kll4\" (UniqueName: \"kubernetes.io/projected/28e91e96-2e4d-4df8-b9fa-928f401194c6-kube-api-access-6kll4\") pod \"28e91e96-2e4d-4df8-b9fa-928f401194c6\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " Oct 30 02:17:17.208883 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.208855 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/679f8047-1bb0-4123-92d7-6474c85ce2be-kube-api-access-c9vw5\" (UniqueName: \"kubernetes.io/projected/679f8047-1bb0-4123-92d7-6474c85ce2be-kube-api-access-c9vw5\") pod \"679f8047-1bb0-4123-92d7-6474c85ce2be\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " Oct 30 02:17:17.208990 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.208909 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-maas-api-tls\") pod \"679f8047-1bb0-4123-92d7-6474c85ce2be\" (UID: \"679f8047-1bb0-4123-92d7-6474c85ce2be\") " Oct 30 02:17:17.208990 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.207236 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1"} err="failed to get container status \"552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1\": rpc error: code = NotFound desc = could not find container \"552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1\": container with ID starting with 552c47452b3601de3e0456c100b69fca02fca731ffc26e74346f732c7a56efb1 not found: ID does not exist" Oct 30 02:17:17.208990 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.208980 5055 scope.go:122] "RemoveContainer" containerID="9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc" Oct 30 02:17:17.209182 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.208957 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-model-cache\") pod \"28e91e96-2e4d-4df8-b9fa-928f401194c6\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " Oct 30 02:17:17.209182 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.209134 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e91e96-2e4d-4df8-b9fa-928f401194c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/28e91e96-2e4d-4df8-b9fa-928f401194c6-tls-certs\") pod \"28e91e96-2e4d-4df8-b9fa-928f401194c6\" (UID: \"28e91e96-2e4d-4df8-b9fa-928f401194c6\") " Oct 30 02:17:17.209289 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.209225 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-model-cache" pod "28e91e96-2e4d-4df8-b9fa-928f401194c6" (UID: "28e91e96-2e4d-4df8-b9fa-928f401194c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:17.209474 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.209450 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-tmp-dir" pod "28e91e96-2e4d-4df8-b9fa-928f401194c6" (UID: "28e91e96-2e4d-4df8-b9fa-928f401194c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:17.209642 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.209617 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/679f8047-1bb0-4123-92d7-6474c85ce2be-openshift-service-ca" pod "679f8047-1bb0-4123-92d7-6474c85ce2be" (UID: "679f8047-1bb0-4123-92d7-6474c85ce2be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:17.210000 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.209976 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.210096 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.210007 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.210096 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.210021 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/679f8047-1bb0-4123-92d7-6474c85ce2be-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.211207 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.211117 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-home" pod "28e91e96-2e4d-4df8-b9fa-928f401194c6" (UID: "28e91e96-2e4d-4df8-b9fa-928f401194c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:17.220345 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.220301 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-metrics-certs" pod "679f8047-1bb0-4123-92d7-6474c85ce2be" (UID: "679f8047-1bb0-4123-92d7-6474c85ce2be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:17.220459 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.220308 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/679f8047-1bb0-4123-92d7-6474c85ce2be-kube-api-access-c9vw5" pod "679f8047-1bb0-4123-92d7-6474c85ce2be" (UID: "679f8047-1bb0-4123-92d7-6474c85ce2be"). InnerVolumeSpecName "kube-api-access-c9vw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:17.220459 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.220445 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e91e96-2e4d-4df8-b9fa-928f401194c6-kube-api-access-6kll4" pod "28e91e96-2e4d-4df8-b9fa-928f401194c6" (UID: "28e91e96-2e4d-4df8-b9fa-928f401194c6"). InnerVolumeSpecName "kube-api-access-6kll4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:17.220599 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.220459 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e91e96-2e4d-4df8-b9fa-928f401194c6-tls-certs" pod "28e91e96-2e4d-4df8-b9fa-928f401194c6" (UID: "28e91e96-2e4d-4df8-b9fa-928f401194c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:17.221450 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.221428 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-dshm" pod "28e91e96-2e4d-4df8-b9fa-928f401194c6" (UID: "28e91e96-2e4d-4df8-b9fa-928f401194c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:17.222586 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.222561 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-maas-api-tls" pod "679f8047-1bb0-4123-92d7-6474c85ce2be" (UID: "679f8047-1bb0-4123-92d7-6474c85ce2be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:17.229542 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.229518 5055 scope.go:122] "RemoveContainer" containerID="9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc" Oct 30 02:17:17.229926 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:17:17.229897 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc\": container with ID starting with 9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc not found: ID does not exist" containerID="9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc" Oct 30 02:17:17.230017 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.229938 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc"} err="failed to get container status \"9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc\": rpc error: code = NotFound desc = could not find container \"9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc\": container with ID starting with 9ad93a7bb947ddc3eecf00ce7f6282eb68a64b1c3b2722f6206eb7f64be1dadc not found: ID does not exist" Oct 30 02:17:17.273313 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.273274 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" podUID="0f1019d4-0665-4023-b5fe-52f5f2394723" containerName="payload-pre-processing" containerID="cri-o://0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00" gracePeriod=25 Oct 30 02:17:17.311462 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.311423 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9vw5\" (UniqueName: \"kubernetes.io/projected/679f8047-1bb0-4123-92d7-6474c85ce2be-kube-api-access-c9vw5\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.311462 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.311462 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.311894 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.311475 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e91e96-2e4d-4df8-b9fa-928f401194c6-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.311894 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.311487 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/679f8047-1bb0-4123-92d7-6474c85ce2be-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.311894 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.311499 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.311894 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.311512 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e91e96-2e4d-4df8-b9fa-928f401194c6-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.311894 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.311524 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kll4\" (UniqueName: \"kubernetes.io/projected/28e91e96-2e4d-4df8-b9fa-928f401194c6-kube-api-access-6kll4\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.531804 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.531743 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm"] Oct 30 02:17:17.543628 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.543594 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-601d4b/test-model-601d4b-kserve-688559f56f-6bcxm"] Oct 30 02:17:17.572737 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.572703 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5"] Oct 30 02:17:17.579998 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:17.579963 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5"] Oct 30 02:17:18.934240 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:18.932940 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e91e96-2e4d-4df8-b9fa-928f401194c6" path="/var/lib/kubelet/pods/28e91e96-2e4d-4df8-b9fa-928f401194c6/volumes" Oct 30 02:17:18.934240 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:18.933734 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="679f8047-1bb0-4123-92d7-6474c85ce2be" path="/var/lib/kubelet/pods/679f8047-1bb0-4123-92d7-6474c85ce2be/volumes" Oct 30 02:17:19.238847 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:17:19.238804 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:17:19.993349 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:19.992446 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f89697-5f5d8b6cbf-f99k5" podUID="679f8047-1bb0-4123-92d7-6474c85ce2be" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.183:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:17:20.178350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:20.176569 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx"] Oct 30 02:17:20.227350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:20.226760 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf"] Oct 30 02:17:20.465523 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:20.465482 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" podUID="0f1019d4-0665-4023-b5fe-52f5f2394723" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:22.016213 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.016174 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln"] Oct 30 02:17:22.016727 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.016458 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" podUID="98ae70a5-1283-439e-9b9d-12644878a758" containerName="maas-api" containerID="cri-o://576a256a2fb1c835a10c0cd736bf682423184a22ee22e55e94b8a979a904ff4c" gracePeriod=30 Oct 30 02:17:22.550246 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.550212 5055 generic.go:356] "Generic (PLEG): container finished" podID="98ae70a5-1283-439e-9b9d-12644878a758" containerID="576a256a2fb1c835a10c0cd736bf682423184a22ee22e55e94b8a979a904ff4c" exitCode=0 Oct 30 02:17:22.550492 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.550465 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" event={"ID":"98ae70a5-1283-439e-9b9d-12644878a758","Type":"ContainerDied","Data":"576a256a2fb1c835a10c0cd736bf682423184a22ee22e55e94b8a979a904ff4c"} Oct 30 02:17:22.620050 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.620023 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:17:22.724728 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.724686 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ae70a5-1283-439e-9b9d-12644878a758-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ae70a5-1283-439e-9b9d-12644878a758-openshift-service-ca\") pod \"98ae70a5-1283-439e-9b9d-12644878a758\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " Oct 30 02:17:22.724728 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.724726 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ae70a5-1283-439e-9b9d-12644878a758-kube-api-access-tkpjm\" (UniqueName: \"kubernetes.io/projected/98ae70a5-1283-439e-9b9d-12644878a758-kube-api-access-tkpjm\") pod \"98ae70a5-1283-439e-9b9d-12644878a758\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " Oct 30 02:17:22.725031 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.724852 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-maas-api-tls\") pod \"98ae70a5-1283-439e-9b9d-12644878a758\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " Oct 30 02:17:22.725031 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.724924 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-metrics-certs\") pod \"98ae70a5-1283-439e-9b9d-12644878a758\" (UID: \"98ae70a5-1283-439e-9b9d-12644878a758\") " Oct 30 02:17:22.725166 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.725127 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ae70a5-1283-439e-9b9d-12644878a758-openshift-service-ca" pod "98ae70a5-1283-439e-9b9d-12644878a758" (UID: "98ae70a5-1283-439e-9b9d-12644878a758"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:22.725715 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.725689 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98ae70a5-1283-439e-9b9d-12644878a758-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:22.728209 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.728164 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-maas-api-tls" pod "98ae70a5-1283-439e-9b9d-12644878a758" (UID: "98ae70a5-1283-439e-9b9d-12644878a758"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:22.728569 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.728490 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ae70a5-1283-439e-9b9d-12644878a758-kube-api-access-tkpjm" pod "98ae70a5-1283-439e-9b9d-12644878a758" (UID: "98ae70a5-1283-439e-9b9d-12644878a758"). InnerVolumeSpecName "kube-api-access-tkpjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:22.728968 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.728931 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-metrics-certs" pod "98ae70a5-1283-439e-9b9d-12644878a758" (UID: "98ae70a5-1283-439e-9b9d-12644878a758"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:22.827061 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.826986 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkpjm\" (UniqueName: \"kubernetes.io/projected/98ae70a5-1283-439e-9b9d-12644878a758-kube-api-access-tkpjm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:22.827061 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.827013 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:22.827061 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:22.827023 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98ae70a5-1283-439e-9b9d-12644878a758-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:23.559747 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:23.559718 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" Oct 30 02:17:23.560222 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:23.559756 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln" event={"ID":"98ae70a5-1283-439e-9b9d-12644878a758","Type":"ContainerDied","Data":"6c493ce3e3699b5895d8bd5f7cdc7ca2cc3b75fae42c79691e9e29c558abc368"} Oct 30 02:17:23.560222 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:23.559820 5055 scope.go:122] "RemoveContainer" containerID="576a256a2fb1c835a10c0cd736bf682423184a22ee22e55e94b8a979a904ff4c" Oct 30 02:17:23.612474 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:23.612440 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln"] Oct 30 02:17:23.625470 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:23.625434 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-22e99d-b8f9bb6b4-qxdln"] Oct 30 02:17:24.840348 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:24.839951 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ae70a5-1283-439e-9b9d-12644878a758" path="/var/lib/kubelet/pods/98ae70a5-1283-439e-9b9d-12644878a758/volumes" Oct 30 02:17:30.465871 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:30.465838 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" podUID="0f1019d4-0665-4023-b5fe-52f5f2394723" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:17:35.250985 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.250938 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" containerID="cri-o://d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7" gracePeriod=45 Oct 30 02:17:35.270290 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.270251 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" containerID="cri-o://aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b" gracePeriod=45 Oct 30 02:17:35.614012 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.613748 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 02:17:35.619413 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.619340 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 02:17:35.651622 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.651583 5055 generic.go:356] "Generic (PLEG): container finished" podID="488e9de5-05b4-48ed-a23a-f4db797631df" containerID="aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b" exitCode=0 Oct 30 02:17:35.651938 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.651916 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" event={"ID":"488e9de5-05b4-48ed-a23a-f4db797631df","Type":"ContainerDied","Data":"aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b"} Oct 30 02:17:35.652072 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.651960 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" event={"ID":"488e9de5-05b4-48ed-a23a-f4db797631df","Type":"ContainerDied","Data":"75b349f848127300b1e61e3f22185971733ea57142da95dce5599fcc9fafd9f3"} Oct 30 02:17:35.652072 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.651983 5055 scope.go:122] "RemoveContainer" containerID="aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b" Oct 30 02:17:35.652212 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.652144 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf" Oct 30 02:17:35.655232 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.655201 5055 generic.go:356] "Generic (PLEG): container finished" podID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerID="d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7" exitCode=0 Oct 30 02:17:35.655431 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.655249 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" event={"ID":"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c","Type":"ContainerDied","Data":"d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7"} Oct 30 02:17:35.655431 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.655282 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" event={"ID":"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c","Type":"ContainerDied","Data":"c7b1a82b3a9ea2d26be6001bf4777bbae60a2d8d45aa3d6971d37fc0c815d1f0"} Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.655466 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx" Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656037 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-home\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-home\") pod \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656287 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-home" pod "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" (UID: "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656422 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/488e9de5-05b4-48ed-a23a-f4db797631df-kube-api-access-wqnnj\" (UniqueName: \"kubernetes.io/projected/488e9de5-05b4-48ed-a23a-f4db797631df-kube-api-access-wqnnj\") pod \"488e9de5-05b4-48ed-a23a-f4db797631df\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656480 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-kube-api-access-6kvhl\" (UniqueName: \"kubernetes.io/projected/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-kube-api-access-6kvhl\") pod \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656558 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tls-certs\") pod \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656609 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-dshm\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-dshm\") pod \"488e9de5-05b4-48ed-a23a-f4db797631df\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656644 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-tmp-dir\") pod \"488e9de5-05b4-48ed-a23a-f4db797631df\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656679 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tmp-dir\") pod \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656744 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/488e9de5-05b4-48ed-a23a-f4db797631df-tls-certs\" (UniqueName: \"kubernetes.io/secret/488e9de5-05b4-48ed-a23a-f4db797631df-tls-certs\") pod \"488e9de5-05b4-48ed-a23a-f4db797631df\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656780 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-home\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-home\") pod \"488e9de5-05b4-48ed-a23a-f4db797631df\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " Oct 30 02:17:35.656983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.656832 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-dshm\") pod \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " Oct 30 02:17:35.661588 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.657812 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-tmp-dir" pod "488e9de5-05b4-48ed-a23a-f4db797631df" (UID: "488e9de5-05b4-48ed-a23a-f4db797631df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.661588 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.658173 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-home" pod "488e9de5-05b4-48ed-a23a-f4db797631df" (UID: "488e9de5-05b4-48ed-a23a-f4db797631df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.661588 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.658225 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tmp-dir" pod "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" (UID: "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.670100 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.669740 5055 scope.go:122] "RemoveContainer" containerID="aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b" Oct 30 02:17:35.671777 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.670373 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-model-cache\") pod \"488e9de5-05b4-48ed-a23a-f4db797631df\" (UID: \"488e9de5-05b4-48ed-a23a-f4db797631df\") " Oct 30 02:17:35.671777 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.670456 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-model-cache\") pod \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\" (UID: \"a120922c-cb9e-4f13-81aa-0b68ccdf0f9c\") " Oct 30 02:17:35.671777 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.671155 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.671777 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.671175 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.671777 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.671190 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.671777 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.671202 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.671777 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.671766 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-model-cache" pod "488e9de5-05b4-48ed-a23a-f4db797631df" (UID: "488e9de5-05b4-48ed-a23a-f4db797631df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.672221 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.671810 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-model-cache" pod "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" (UID: "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.675666 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:17:35.673443 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b\": container with ID starting with aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b not found: ID does not exist" containerID="aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b" Oct 30 02:17:35.675666 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.673486 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b"} err="failed to get container status \"aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b\": rpc error: code = NotFound desc = could not find container \"aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b\": container with ID starting with aeb36035903a4582b53840af72cdcce750d92bd135be7b16eb08a65880c6bd4b not found: ID does not exist" Oct 30 02:17:35.675666 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.673522 5055 scope.go:122] "RemoveContainer" containerID="d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7" Oct 30 02:17:35.689864 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.689801 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/488e9de5-05b4-48ed-a23a-f4db797631df-kube-api-access-wqnnj" pod "488e9de5-05b4-48ed-a23a-f4db797631df" (UID: "488e9de5-05b4-48ed-a23a-f4db797631df"). InnerVolumeSpecName "kube-api-access-wqnnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:35.690932 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.690907 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tls-certs" pod "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" (UID: "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:35.695850 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.695824 5055 scope.go:122] "RemoveContainer" containerID="d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7" Oct 30 02:17:35.696505 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:17:35.696475 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7\": container with ID starting with d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7 not found: ID does not exist" containerID="d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7" Oct 30 02:17:35.696608 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.696516 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7"} err="failed to get container status \"d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7\": rpc error: code = NotFound desc = could not find container \"d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7\": container with ID starting with d997249ec238f8fff1ba88c721e053e3620feb3b4e23d96e1492e6dd03c063f7 not found: ID does not exist" Oct 30 02:17:35.707869 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.707815 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-dshm" pod "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" (UID: "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.708349 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.708186 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-kube-api-access-6kvhl" pod "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" (UID: "a120922c-cb9e-4f13-81aa-0b68ccdf0f9c"). InnerVolumeSpecName "kube-api-access-6kvhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:35.708349 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.708297 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/488e9de5-05b4-48ed-a23a-f4db797631df-tls-certs" pod "488e9de5-05b4-48ed-a23a-f4db797631df" (UID: "488e9de5-05b4-48ed-a23a-f4db797631df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:35.710132 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.710103 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-dshm" pod "488e9de5-05b4-48ed-a23a-f4db797631df" (UID: "488e9de5-05b4-48ed-a23a-f4db797631df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.787358 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.782855 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/488e9de5-05b4-48ed-a23a-f4db797631df-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.787358 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.782891 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.787358 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.782904 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.787358 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.782916 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.787358 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.782938 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqnnj\" (UniqueName: \"kubernetes.io/projected/488e9de5-05b4-48ed-a23a-f4db797631df-kube-api-access-wqnnj\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.787358 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.782952 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvhl\" (UniqueName: \"kubernetes.io/projected/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-kube-api-access-6kvhl\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.787358 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.782970 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.787358 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:35.782987 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/488e9de5-05b4-48ed-a23a-f4db797631df-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:36.004637 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:36.004598 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx"] Oct 30 02:17:36.017012 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:36.016971 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/premium-w2-eda881-kserve-845596dd65-9dmmx"] Oct 30 02:17:36.043611 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:36.043570 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf"] Oct 30 02:17:36.047512 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:36.047473 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-eda881/simulator-w2-eda881-kserve-6cc464684d-4cmtf"] Oct 30 02:17:36.820517 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:36.820475 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" path="/var/lib/kubelet/pods/488e9de5-05b4-48ed-a23a-f4db797631df/volumes" Oct 30 02:17:36.821234 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:36.821210 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" path="/var/lib/kubelet/pods/a120922c-cb9e-4f13-81aa-0b68ccdf0f9c/volumes" Oct 30 02:17:37.573579 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.573547 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:17:37.607936 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.607892 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f1019d4-0665-4023-b5fe-52f5f2394723-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1019d4-0665-4023-b5fe-52f5f2394723-plugins-config-volume\") pod \"0f1019d4-0665-4023-b5fe-52f5f2394723\" (UID: \"0f1019d4-0665-4023-b5fe-52f5f2394723\") " Oct 30 02:17:37.608126 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.608047 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1019d4-0665-4023-b5fe-52f5f2394723-kube-api-access-lk4zn\" (UniqueName: \"kubernetes.io/projected/0f1019d4-0665-4023-b5fe-52f5f2394723-kube-api-access-lk4zn\") pod \"0f1019d4-0665-4023-b5fe-52f5f2394723\" (UID: \"0f1019d4-0665-4023-b5fe-52f5f2394723\") " Oct 30 02:17:37.608395 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.608369 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1019d4-0665-4023-b5fe-52f5f2394723-plugins-config-volume" pod "0f1019d4-0665-4023-b5fe-52f5f2394723" (UID: "0f1019d4-0665-4023-b5fe-52f5f2394723"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:37.608645 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.608626 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1019d4-0665-4023-b5fe-52f5f2394723-plugins-config-volume\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:37.610796 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.610769 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1019d4-0665-4023-b5fe-52f5f2394723-kube-api-access-lk4zn" pod "0f1019d4-0665-4023-b5fe-52f5f2394723" (UID: "0f1019d4-0665-4023-b5fe-52f5f2394723"). InnerVolumeSpecName "kube-api-access-lk4zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:37.679222 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.674627 5055 generic.go:356] "Generic (PLEG): container finished" podID="0f1019d4-0665-4023-b5fe-52f5f2394723" containerID="0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00" exitCode=0 Oct 30 02:17:37.679222 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.674668 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" event={"ID":"0f1019d4-0665-4023-b5fe-52f5f2394723","Type":"ContainerDied","Data":"0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00"} Oct 30 02:17:37.679222 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.674715 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" Oct 30 02:17:37.679222 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.674738 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5" event={"ID":"0f1019d4-0665-4023-b5fe-52f5f2394723","Type":"ContainerDied","Data":"af295f337a8bf1854b7f983f27d29f3413c021323789dcca1e2fec610d5e8ec7"} Oct 30 02:17:37.679222 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.674768 5055 scope.go:122] "RemoveContainer" containerID="0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00" Oct 30 02:17:37.689298 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.689271 5055 scope.go:122] "RemoveContainer" containerID="0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00" Oct 30 02:17:37.689766 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:17:37.689742 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00\": container with ID starting with 0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00 not found: ID does not exist" containerID="0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00" Oct 30 02:17:37.689837 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.689781 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00"} err="failed to get container status \"0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00\": rpc error: code = NotFound desc = could not find container \"0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00\": container with ID starting with 0a90ea2f50e46b836f055fe0caa2996f7fbcf159517188379f4f9de0286f1e00 not found: ID does not exist" Oct 30 02:17:37.710793 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.710745 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk4zn\" (UniqueName: \"kubernetes.io/projected/0f1019d4-0665-4023-b5fe-52f5f2394723-kube-api-access-lk4zn\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:37.737762 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.732572 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5"] Oct 30 02:17:37.737994 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:37.737830 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f89697-77bdcfcd99-5r7z5"] Oct 30 02:17:38.821943 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:38.821906 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1019d4-0665-4023-b5fe-52f5f2394723" path="/var/lib/kubelet/pods/0f1019d4-0665-4023-b5fe-52f5f2394723/volumes" Oct 30 02:17:47.243288 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:47.243248 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2"] Oct 30 02:17:47.243912 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:47.243508 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" podUID="dec73c66-91e7-4f40-9a01-0080db07ec87" containerName="maas-api" containerID="cri-o://06b1653b2aa411bc4da7df8b635dfedbe395e168cc07f80ac401eb873026079a" gracePeriod=30 Oct 30 02:17:47.611407 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:47.606599 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p"] Oct 30 02:17:47.611407 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:47.606862 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" podUID="7baef220-6084-416e-a7c2-c86722049b2f" containerName="maas-api" containerID="cri-o://ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc" gracePeriod=30 Oct 30 02:17:47.839730 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:47.838960 5055 generic.go:356] "Generic (PLEG): container finished" podID="dec73c66-91e7-4f40-9a01-0080db07ec87" containerID="06b1653b2aa411bc4da7df8b635dfedbe395e168cc07f80ac401eb873026079a" exitCode=0 Oct 30 02:17:47.839730 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:47.839121 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" event={"ID":"dec73c66-91e7-4f40-9a01-0080db07ec87","Type":"ContainerDied","Data":"06b1653b2aa411bc4da7df8b635dfedbe395e168cc07f80ac401eb873026079a"} Oct 30 02:17:48.007379 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.006738 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:17:48.101368 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.093228 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-maas-api-tls\") pod \"dec73c66-91e7-4f40-9a01-0080db07ec87\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " Oct 30 02:17:48.101368 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.093289 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dec73c66-91e7-4f40-9a01-0080db07ec87-kube-api-access-d6cbk\" (UniqueName: \"kubernetes.io/projected/dec73c66-91e7-4f40-9a01-0080db07ec87-kube-api-access-d6cbk\") pod \"dec73c66-91e7-4f40-9a01-0080db07ec87\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " Oct 30 02:17:48.101368 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.093395 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dec73c66-91e7-4f40-9a01-0080db07ec87-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec73c66-91e7-4f40-9a01-0080db07ec87-openshift-service-ca\") pod \"dec73c66-91e7-4f40-9a01-0080db07ec87\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " Oct 30 02:17:48.101368 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.093604 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-metrics-certs\") pod \"dec73c66-91e7-4f40-9a01-0080db07ec87\" (UID: \"dec73c66-91e7-4f40-9a01-0080db07ec87\") " Oct 30 02:17:48.101368 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.095251 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dec73c66-91e7-4f40-9a01-0080db07ec87-openshift-service-ca" pod "dec73c66-91e7-4f40-9a01-0080db07ec87" (UID: "dec73c66-91e7-4f40-9a01-0080db07ec87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:48.104823 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.104790 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec73c66-91e7-4f40-9a01-0080db07ec87-kube-api-access-d6cbk" pod "dec73c66-91e7-4f40-9a01-0080db07ec87" (UID: "dec73c66-91e7-4f40-9a01-0080db07ec87"). InnerVolumeSpecName "kube-api-access-d6cbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:48.108022 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.107882 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-metrics-certs" pod "dec73c66-91e7-4f40-9a01-0080db07ec87" (UID: "dec73c66-91e7-4f40-9a01-0080db07ec87"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:48.112190 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.112163 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-maas-api-tls" pod "dec73c66-91e7-4f40-9a01-0080db07ec87" (UID: "dec73c66-91e7-4f40-9a01-0080db07ec87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:48.133675 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.133644 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 02:17:48.231350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.218855 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-maas-api-tls\") pod \"7baef220-6084-416e-a7c2-c86722049b2f\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " Oct 30 02:17:48.231350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.218928 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7baef220-6084-416e-a7c2-c86722049b2f-kube-api-access-6ww7p\" (UniqueName: \"kubernetes.io/projected/7baef220-6084-416e-a7c2-c86722049b2f-kube-api-access-6ww7p\") pod \"7baef220-6084-416e-a7c2-c86722049b2f\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " Oct 30 02:17:48.231350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.218973 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7baef220-6084-416e-a7c2-c86722049b2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7baef220-6084-416e-a7c2-c86722049b2f-openshift-service-ca\") pod \"7baef220-6084-416e-a7c2-c86722049b2f\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " Oct 30 02:17:48.231350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.219166 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-metrics-certs\") pod \"7baef220-6084-416e-a7c2-c86722049b2f\" (UID: \"7baef220-6084-416e-a7c2-c86722049b2f\") " Oct 30 02:17:48.231350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.219704 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:48.231350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.219726 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6cbk\" (UniqueName: \"kubernetes.io/projected/dec73c66-91e7-4f40-9a01-0080db07ec87-kube-api-access-d6cbk\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:48.231350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.219739 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dec73c66-91e7-4f40-9a01-0080db07ec87-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:48.231350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.219752 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dec73c66-91e7-4f40-9a01-0080db07ec87-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:48.231350 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.226578 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7baef220-6084-416e-a7c2-c86722049b2f-openshift-service-ca" pod "7baef220-6084-416e-a7c2-c86722049b2f" (UID: "7baef220-6084-416e-a7c2-c86722049b2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:48.286948 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.286445 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7baef220-6084-416e-a7c2-c86722049b2f-kube-api-access-6ww7p" pod "7baef220-6084-416e-a7c2-c86722049b2f" (UID: "7baef220-6084-416e-a7c2-c86722049b2f"). InnerVolumeSpecName "kube-api-access-6ww7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:48.307573 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.299493 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-maas-api-tls" pod "7baef220-6084-416e-a7c2-c86722049b2f" (UID: "7baef220-6084-416e-a7c2-c86722049b2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:48.325280 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.323867 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:48.325280 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.323894 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ww7p\" (UniqueName: \"kubernetes.io/projected/7baef220-6084-416e-a7c2-c86722049b2f-kube-api-access-6ww7p\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:48.325280 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.323905 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7baef220-6084-416e-a7c2-c86722049b2f-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:48.334206 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.334148 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-metrics-certs" pod "7baef220-6084-416e-a7c2-c86722049b2f" (UID: "7baef220-6084-416e-a7c2-c86722049b2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:48.425132 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.425089 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7baef220-6084-416e-a7c2-c86722049b2f-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:17:48.847526 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.847495 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" Oct 30 02:17:48.847526 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.847520 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2" event={"ID":"dec73c66-91e7-4f40-9a01-0080db07ec87","Type":"ContainerDied","Data":"b3b9864d500d2a2909a02f3caef09ddbd07928c55f048ac046c6c45f1f7b7c06"} Oct 30 02:17:48.847800 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.847595 5055 scope.go:122] "RemoveContainer" containerID="06b1653b2aa411bc4da7df8b635dfedbe395e168cc07f80ac401eb873026079a" Oct 30 02:17:48.850882 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.850855 5055 generic.go:356] "Generic (PLEG): container finished" podID="7baef220-6084-416e-a7c2-c86722049b2f" containerID="ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc" exitCode=0 Oct 30 02:17:48.850997 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.850969 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" event={"ID":"7baef220-6084-416e-a7c2-c86722049b2f","Type":"ContainerDied","Data":"ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc"} Oct 30 02:17:48.851038 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.851013 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" event={"ID":"7baef220-6084-416e-a7c2-c86722049b2f","Type":"ContainerDied","Data":"e957905b5b1a89cb9f07b97efa7a042a8e53b6654ea133bb4c77e546792061a8"} Oct 30 02:17:48.851094 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.851079 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p" Oct 30 02:17:48.859296 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.859266 5055 scope.go:122] "RemoveContainer" containerID="ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc" Oct 30 02:17:48.872446 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.872417 5055 scope.go:122] "RemoveContainer" containerID="ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc" Oct 30 02:17:48.872870 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:17:48.872840 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc\": container with ID starting with ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc not found: ID does not exist" containerID="ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc" Oct 30 02:17:48.872983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.872884 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc"} err="failed to get container status \"ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc\": rpc error: code = NotFound desc = could not find container \"ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc\": container with ID starting with ef8817f157a2aa8367cbc9ba470ba5e3da725fb47a86b62ac71dfddebcfe03dc not found: ID does not exist" Oct 30 02:17:48.901439 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.901380 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p"] Oct 30 02:17:48.914026 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.913993 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-d44413-74ff68c7cb-r9r5p"] Oct 30 02:17:48.941093 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.941054 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2"] Oct 30 02:17:48.944766 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:48.944741 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-601d4b-65f6f49b7d-7rpl2"] Oct 30 02:17:49.220526 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:17:49.220493 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:17:50.820967 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:50.820930 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7baef220-6084-416e-a7c2-c86722049b2f" path="/var/lib/kubelet/pods/7baef220-6084-416e-a7c2-c86722049b2f/volumes" Oct 30 02:17:50.821638 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:50.821617 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dec73c66-91e7-4f40-9a01-0080db07ec87" path="/var/lib/kubelet/pods/dec73c66-91e7-4f40-9a01-0080db07ec87/volumes" Oct 30 02:17:55.102806 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:55.102707 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh"] Oct 30 02:17:55.103273 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:55.102966 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" podUID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" containerName="istio-proxy" containerID="cri-o://55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c" gracePeriod=30 Oct 30 02:17:55.107743 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:55.107709 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn"] Oct 30 02:17:55.108030 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:55.107968 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerName="istio-proxy" containerID="cri-o://200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007" gracePeriod=30 Oct 30 02:17:55.707899 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:55.707835 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5"] Oct 30 02:17:55.708699 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:17:55.708395 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" containerID="cri-o://66cc112c59ed1e4d6659c8499f9f41b286f726be3aff3da77d0193bef280a496" gracePeriod=30 Oct 30 02:18:00.426963 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.426931 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.454830 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-token\" (UniqueName: \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-token\") pod \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.454890 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-socket\") pod \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.454982 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-podinfo\") pod \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.455028 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-credential-socket\") pod \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.455070 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-certs\") pod \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.455133 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istiod-ca-cert\") pod \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.455183 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-envoy\") pod \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.455238 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-data\") pod \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.455410 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-kube-api-access-7gzj2\" (UniqueName: \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-kube-api-access-7gzj2\") pod \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\" (UID: \"a0b9ecdc-93c7-4284-9c50-92ce07e34d90\") " Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.455763 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-credential-socket" pod "a0b9ecdc-93c7-4284-9c50-92ce07e34d90" (UID: "a0b9ecdc-93c7-4284-9c50-92ce07e34d90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.456150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.456038 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.457430 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.457384 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-socket" pod "a0b9ecdc-93c7-4284-9c50-92ce07e34d90" (UID: "a0b9ecdc-93c7-4284-9c50-92ce07e34d90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.457752 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.457668 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-certs" pod "a0b9ecdc-93c7-4284-9c50-92ce07e34d90" (UID: "a0b9ecdc-93c7-4284-9c50-92ce07e34d90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.457752 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.457731 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-data" pod "a0b9ecdc-93c7-4284-9c50-92ce07e34d90" (UID: "a0b9ecdc-93c7-4284-9c50-92ce07e34d90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.457951 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.457873 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istiod-ca-cert" pod "a0b9ecdc-93c7-4284-9c50-92ce07e34d90" (UID: "a0b9ecdc-93c7-4284-9c50-92ce07e34d90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:00.458669 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.458643 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-kube-api-access-7gzj2" pod "a0b9ecdc-93c7-4284-9c50-92ce07e34d90" (UID: "a0b9ecdc-93c7-4284-9c50-92ce07e34d90"). InnerVolumeSpecName "kube-api-access-7gzj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:00.459907 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.459872 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-envoy" pod "a0b9ecdc-93c7-4284-9c50-92ce07e34d90" (UID: "a0b9ecdc-93c7-4284-9c50-92ce07e34d90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.484033 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.473625 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-token" pod "a0b9ecdc-93c7-4284-9c50-92ce07e34d90" (UID: "a0b9ecdc-93c7-4284-9c50-92ce07e34d90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:00.484033 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.474582 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-podinfo" pod "a0b9ecdc-93c7-4284-9c50-92ce07e34d90" (UID: "a0b9ecdc-93c7-4284-9c50-92ce07e34d90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:00.502686 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.502650 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:18:00.557143 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557112 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-socket\") pod \"a0dfd823-2436-423a-b2f6-67a537b6a329\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " Oct 30 02:18:00.557351 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557169 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-credential-socket\") pod \"a0dfd823-2436-423a-b2f6-67a537b6a329\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " Oct 30 02:18:00.557351 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557203 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-token\" (UniqueName: \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-token\") pod \"a0dfd823-2436-423a-b2f6-67a537b6a329\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " Oct 30 02:18:00.557351 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557232 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-podinfo\") pod \"a0dfd823-2436-423a-b2f6-67a537b6a329\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " Oct 30 02:18:00.557532 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557509 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-data\") pod \"a0dfd823-2436-423a-b2f6-67a537b6a329\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " Oct 30 02:18:00.557571 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557503 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-socket" pod "a0dfd823-2436-423a-b2f6-67a537b6a329" (UID: "a0dfd823-2436-423a-b2f6-67a537b6a329"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.557607 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557569 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-kube-api-access-txbwd\" (UniqueName: \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-kube-api-access-txbwd\") pod \"a0dfd823-2436-423a-b2f6-67a537b6a329\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " Oct 30 02:18:00.557607 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557566 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-credential-socket" pod "a0dfd823-2436-423a-b2f6-67a537b6a329" (UID: "a0dfd823-2436-423a-b2f6-67a537b6a329"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.557607 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557590 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-envoy\") pod \"a0dfd823-2436-423a-b2f6-67a537b6a329\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " Oct 30 02:18:00.557703 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557614 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-certs\") pod \"a0dfd823-2436-423a-b2f6-67a537b6a329\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " Oct 30 02:18:00.557703 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.557666 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0dfd823-2436-423a-b2f6-67a537b6a329-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dfd823-2436-423a-b2f6-67a537b6a329-istiod-ca-cert\") pod \"a0dfd823-2436-423a-b2f6-67a537b6a329\" (UID: \"a0dfd823-2436-423a-b2f6-67a537b6a329\") " Oct 30 02:18:00.558053 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558040 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.558129 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558057 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.558129 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558067 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.558129 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558077 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.558129 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558085 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.558129 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558092 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.558129 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558100 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gzj2\" (UniqueName: \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-kube-api-access-7gzj2\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.558129 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558109 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.558129 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558118 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.558129 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558127 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0b9ecdc-93c7-4284-9c50-92ce07e34d90-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.558554 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558360 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-data" pod "a0dfd823-2436-423a-b2f6-67a537b6a329" (UID: "a0dfd823-2436-423a-b2f6-67a537b6a329"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.558742 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558717 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-certs" pod "a0dfd823-2436-423a-b2f6-67a537b6a329" (UID: "a0dfd823-2436-423a-b2f6-67a537b6a329"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.558831 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.558775 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0dfd823-2436-423a-b2f6-67a537b6a329-istiod-ca-cert" pod "a0dfd823-2436-423a-b2f6-67a537b6a329" (UID: "a0dfd823-2436-423a-b2f6-67a537b6a329"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:00.560721 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.560682 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-podinfo" pod "a0dfd823-2436-423a-b2f6-67a537b6a329" (UID: "a0dfd823-2436-423a-b2f6-67a537b6a329"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:00.560849 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.560825 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-token" pod "a0dfd823-2436-423a-b2f6-67a537b6a329" (UID: "a0dfd823-2436-423a-b2f6-67a537b6a329"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:00.560947 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.560929 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-kube-api-access-txbwd" pod "a0dfd823-2436-423a-b2f6-67a537b6a329" (UID: "a0dfd823-2436-423a-b2f6-67a537b6a329"). InnerVolumeSpecName "kube-api-access-txbwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:00.561197 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.561175 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-envoy" pod "a0dfd823-2436-423a-b2f6-67a537b6a329" (UID: "a0dfd823-2436-423a-b2f6-67a537b6a329"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.660003 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.659913 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.660003 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.659946 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0dfd823-2436-423a-b2f6-67a537b6a329-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.660003 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.659955 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.660003 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.659965 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.660003 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.659972 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.660003 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.659982 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txbwd\" (UniqueName: \"kubernetes.io/projected/a0dfd823-2436-423a-b2f6-67a537b6a329-kube-api-access-txbwd\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.660003 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.659993 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0dfd823-2436-423a-b2f6-67a537b6a329-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.962487 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.962458 5055 generic.go:356] "Generic (PLEG): container finished" podID="46824cf2-1354-4205-8a62-21f124771343" containerID="66cc112c59ed1e4d6659c8499f9f41b286f726be3aff3da77d0193bef280a496" exitCode=0 Oct 30 02:18:00.962738 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.962715 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" event={"ID":"46824cf2-1354-4205-8a62-21f124771343","Type":"ContainerDied","Data":"66cc112c59ed1e4d6659c8499f9f41b286f726be3aff3da77d0193bef280a496"} Oct 30 02:18:00.962927 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.962782 5055 scope.go:122] "RemoveContainer" containerID="8b9ed2b5fd4ec8aa25257cad4f66ead27e9cb0c96e523a372eea5b1721c752d0" Oct 30 02:18:00.966973 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.966948 5055 generic.go:356] "Generic (PLEG): container finished" podID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" containerID="55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c" exitCode=0 Oct 30 02:18:00.967223 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.967175 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" event={"ID":"a0b9ecdc-93c7-4284-9c50-92ce07e34d90","Type":"ContainerDied","Data":"55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c"} Oct 30 02:18:00.967497 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.967239 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" event={"ID":"a0b9ecdc-93c7-4284-9c50-92ce07e34d90","Type":"ContainerDied","Data":"2532d3597cab5134dce7292f4049dfd35dd13189a6dc6f65c5a47bf92fd0129d"} Oct 30 02:18:00.967497 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.967343 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh" Oct 30 02:18:00.970608 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.970588 5055 generic.go:356] "Generic (PLEG): container finished" podID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerID="200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007" exitCode=0 Oct 30 02:18:00.970798 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.970783 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" event={"ID":"a0dfd823-2436-423a-b2f6-67a537b6a329","Type":"ContainerDied","Data":"200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007"} Oct 30 02:18:00.972655 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.972632 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" event={"ID":"a0dfd823-2436-423a-b2f6-67a537b6a329","Type":"ContainerDied","Data":"952de181e08d701e5a4515d731b64d7890e15beb1dba2afdb7993276862a2735"} Oct 30 02:18:00.972857 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.972843 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn" Oct 30 02:18:00.992872 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:00.992839 5055 scope.go:122] "RemoveContainer" containerID="55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c" Oct 30 02:18:01.004525 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.004498 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 02:18:01.005862 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.005843 5055 scope.go:122] "RemoveContainer" containerID="55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c" Oct 30 02:18:01.006183 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:01.006162 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c\": container with ID starting with 55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c not found: ID does not exist" containerID="55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c" Oct 30 02:18:01.006256 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.006190 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c"} err="failed to get container status \"55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c\": rpc error: code = NotFound desc = could not find container \"55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c\": container with ID starting with 55146bfc0000a68f995e04cb4ef710cbdcff04f3ab334371af712bd787294f2c not found: ID does not exist" Oct 30 02:18:01.006256 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.006207 5055 scope.go:122] "RemoveContainer" containerID="200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007" Oct 30 02:18:01.016692 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.016666 5055 scope.go:122] "RemoveContainer" containerID="200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007" Oct 30 02:18:01.017000 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:01.016974 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007\": container with ID starting with 200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007 not found: ID does not exist" containerID="200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007" Oct 30 02:18:01.017070 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.017009 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007"} err="failed to get container status \"200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007\": rpc error: code = NotFound desc = could not find container \"200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007\": container with ID starting with 200ff3085b589668b4705bd08193d930f1c118f04cded97e3200dbc854a42007 not found: ID does not exist" Oct 30 02:18:01.028277 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.028247 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn"] Oct 30 02:18:01.040048 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.040014 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-gkbkn"] Oct 30 02:18:01.068077 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.068037 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh"] Oct 30 02:18:01.071781 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.071747 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-dsnwh"] Oct 30 02:18:01.075236 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075211 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46824cf2-1354-4205-8a62-21f124771343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46824cf2-1354-4205-8a62-21f124771343-istio-podinfo\") pod \"46824cf2-1354-4205-8a62-21f124771343\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " Oct 30 02:18:01.075387 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075271 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-data\") pod \"46824cf2-1354-4205-8a62-21f124771343\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " Oct 30 02:18:01.075387 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075303 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-credential-socket\") pod \"46824cf2-1354-4205-8a62-21f124771343\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " Oct 30 02:18:01.075387 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075383 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46824cf2-1354-4205-8a62-21f124771343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46824cf2-1354-4205-8a62-21f124771343-istiod-ca-cert\") pod \"46824cf2-1354-4205-8a62-21f124771343\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " Oct 30 02:18:01.075569 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075488 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-kube-api-access-sbczd\" (UniqueName: \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-kube-api-access-sbczd\") pod \"46824cf2-1354-4205-8a62-21f124771343\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " Oct 30 02:18:01.075569 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075525 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-certs\") pod \"46824cf2-1354-4205-8a62-21f124771343\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " Oct 30 02:18:01.075569 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075555 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-istio-token\" (UniqueName: \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-istio-token\") pod \"46824cf2-1354-4205-8a62-21f124771343\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " Oct 30 02:18:01.075731 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075582 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-envoy\") pod \"46824cf2-1354-4205-8a62-21f124771343\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " Oct 30 02:18:01.075731 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075622 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-socket\") pod \"46824cf2-1354-4205-8a62-21f124771343\" (UID: \"46824cf2-1354-4205-8a62-21f124771343\") " Oct 30 02:18:01.075731 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075683 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-data" pod "46824cf2-1354-4205-8a62-21f124771343" (UID: "46824cf2-1354-4205-8a62-21f124771343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:01.075731 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075719 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-credential-socket" pod "46824cf2-1354-4205-8a62-21f124771343" (UID: "46824cf2-1354-4205-8a62-21f124771343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:01.075958 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.075918 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-certs" pod "46824cf2-1354-4205-8a62-21f124771343" (UID: "46824cf2-1354-4205-8a62-21f124771343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:01.076168 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.076144 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-socket" pod "46824cf2-1354-4205-8a62-21f124771343" (UID: "46824cf2-1354-4205-8a62-21f124771343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:01.076276 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.076189 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.076276 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.076210 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.076276 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.076222 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.077567 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.077540 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46824cf2-1354-4205-8a62-21f124771343-istiod-ca-cert" pod "46824cf2-1354-4205-8a62-21f124771343" (UID: "46824cf2-1354-4205-8a62-21f124771343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:01.078392 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.078369 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46824cf2-1354-4205-8a62-21f124771343-istio-podinfo" pod "46824cf2-1354-4205-8a62-21f124771343" (UID: "46824cf2-1354-4205-8a62-21f124771343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:01.078491 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.078450 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-envoy" pod "46824cf2-1354-4205-8a62-21f124771343" (UID: "46824cf2-1354-4205-8a62-21f124771343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:01.078491 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.078462 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-istio-token" pod "46824cf2-1354-4205-8a62-21f124771343" (UID: "46824cf2-1354-4205-8a62-21f124771343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:01.080461 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.080443 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-kube-api-access-sbczd" pod "46824cf2-1354-4205-8a62-21f124771343" (UID: "46824cf2-1354-4205-8a62-21f124771343"). InnerVolumeSpecName "kube-api-access-sbczd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:01.177776 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.177733 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46824cf2-1354-4205-8a62-21f124771343-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.177776 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.177774 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46824cf2-1354-4205-8a62-21f124771343-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.178013 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.177790 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbczd\" (UniqueName: \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-kube-api-access-sbczd\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.178013 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.177804 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46824cf2-1354-4205-8a62-21f124771343-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.178013 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.177819 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.178013 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.177832 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46824cf2-1354-4205-8a62-21f124771343-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.978840 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.978805 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" event={"ID":"46824cf2-1354-4205-8a62-21f124771343","Type":"ContainerDied","Data":"15ec7e77af5f1f9702eebf5d741c84faeeda3e24f09ff1c0c7d53fdbebcc4f6d"} Oct 30 02:18:01.978840 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.978823 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5" Oct 30 02:18:01.979504 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:01.978876 5055 scope.go:122] "RemoveContainer" containerID="66cc112c59ed1e4d6659c8499f9f41b286f726be3aff3da77d0193bef280a496" Oct 30 02:18:02.031051 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:02.031013 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5"] Oct 30 02:18:02.043507 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:02.043473 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d44413-openshift-default-6df58f8989-x7gw5"] Oct 30 02:18:02.822004 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:02.821970 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46824cf2-1354-4205-8a62-21f124771343" path="/var/lib/kubelet/pods/46824cf2-1354-4205-8a62-21f124771343/volumes" Oct 30 02:18:02.822489 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:02.822473 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" path="/var/lib/kubelet/pods/a0b9ecdc-93c7-4284-9c50-92ce07e34d90/volumes" Oct 30 02:18:02.822913 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:02.822899 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" path="/var/lib/kubelet/pods/a0dfd823-2436-423a-b2f6-67a537b6a329/volumes" Oct 30 02:18:06.119463 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:06.119426 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk"] Oct 30 02:18:06.149752 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:06.149717 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw"] Oct 30 02:18:11.120711 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:11.120666 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" podUID="65f34b67-7d82-405f-8825-8fede2564631" containerName="payload-processing" containerID="cri-o://ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99" gracePeriod=25 Oct 30 02:18:11.150084 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:11.150032 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" podUID="1ae14669-6647-4386-a4fd-7a769705984e" containerName="payload-pre-processing" containerID="cri-o://061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8" gracePeriod=25 Oct 30 02:18:13.487468 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:13.487162 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w"] Oct 30 02:18:13.490637 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:13.487584 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" podUID="e7f5d35f-45a0-4e49-aa98-65836e5d82b5" containerName="maas-api" containerID="cri-o://ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2" gracePeriod=30 Oct 30 02:18:13.948033 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:13.948003 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 02:18:14.046081 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.046033 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-metrics-certs\") pod \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " Oct 30 02:18:14.046309 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.046183 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-openshift-service-ca\") pod \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " Oct 30 02:18:14.046309 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.046243 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-maas-api-tls\") pod \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " Oct 30 02:18:14.046309 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.046276 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-kube-api-access-xf4zb\" (UniqueName: \"kubernetes.io/projected/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-kube-api-access-xf4zb\") pod \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\" (UID: \"e7f5d35f-45a0-4e49-aa98-65836e5d82b5\") " Oct 30 02:18:14.046672 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.046640 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-openshift-service-ca" pod "e7f5d35f-45a0-4e49-aa98-65836e5d82b5" (UID: "e7f5d35f-45a0-4e49-aa98-65836e5d82b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:14.046887 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.046868 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:14.048850 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.048823 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-kube-api-access-xf4zb" pod "e7f5d35f-45a0-4e49-aa98-65836e5d82b5" (UID: "e7f5d35f-45a0-4e49-aa98-65836e5d82b5"). InnerVolumeSpecName "kube-api-access-xf4zb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:14.048945 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.048841 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-metrics-certs" pod "e7f5d35f-45a0-4e49-aa98-65836e5d82b5" (UID: "e7f5d35f-45a0-4e49-aa98-65836e5d82b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:14.049096 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.049072 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-maas-api-tls" pod "e7f5d35f-45a0-4e49-aa98-65836e5d82b5" (UID: "e7f5d35f-45a0-4e49-aa98-65836e5d82b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:14.097407 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.097370 5055 generic.go:356] "Generic (PLEG): container finished" podID="e7f5d35f-45a0-4e49-aa98-65836e5d82b5" containerID="ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2" exitCode=0 Oct 30 02:18:14.097619 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.097469 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" Oct 30 02:18:14.097619 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.097492 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" event={"ID":"e7f5d35f-45a0-4e49-aa98-65836e5d82b5","Type":"ContainerDied","Data":"ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2"} Oct 30 02:18:14.097619 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.097602 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w" event={"ID":"e7f5d35f-45a0-4e49-aa98-65836e5d82b5","Type":"ContainerDied","Data":"c2fed955c96bcc4d4edddd91e7aea262db46c2e740b4899034bbd74093cb3482"} Oct 30 02:18:14.097751 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.097634 5055 scope.go:122] "RemoveContainer" containerID="ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2" Oct 30 02:18:14.123069 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.123044 5055 scope.go:122] "RemoveContainer" containerID="ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2" Oct 30 02:18:14.123621 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:14.123538 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2\": container with ID starting with ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2 not found: ID does not exist" containerID="ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2" Oct 30 02:18:14.123621 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.123578 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2"} err="failed to get container status \"ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2\": rpc error: code = NotFound desc = could not find container \"ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2\": container with ID starting with ec6e5a4855dc6a91ab69cfea1a720c12718da76e583d08219b29e67300275dd2 not found: ID does not exist" Oct 30 02:18:14.147808 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.147770 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:14.148695 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.147809 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf4zb\" (UniqueName: \"kubernetes.io/projected/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-kube-api-access-xf4zb\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:14.148695 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.147928 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7f5d35f-45a0-4e49-aa98-65836e5d82b5-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:14.151532 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.151499 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w"] Oct 30 02:18:14.155736 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.155712 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-76a19e-5967f87589-lxb9w"] Oct 30 02:18:14.821239 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:14.821201 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f5d35f-45a0-4e49-aa98-65836e5d82b5" path="/var/lib/kubelet/pods/e7f5d35f-45a0-4e49-aa98-65836e5d82b5/volumes" Oct 30 02:18:19.222450 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:19.222407 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:18:19.330467 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:19.330433 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx"] Oct 30 02:18:19.330664 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:19.330623 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerName="istio-proxy" containerID="cri-o://b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe" gracePeriod=30 Oct 30 02:18:19.950203 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:19.950164 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" podUID="1ae14669-6647-4386-a4fd-7a769705984e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:19.950432 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:19.950204 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" podUID="65f34b67-7d82-405f-8825-8fede2564631" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:21.390374 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:21.389143 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k"] Oct 30 02:18:21.390374 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:21.389471 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" containerID="cri-o://5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1" gracePeriod=30 Oct 30 02:18:24.628856 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.628822 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 02:18:24.701734 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.701693 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-data\") pod \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " Oct 30 02:18:24.701950 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.701767 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istiod-ca-cert\") pod \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " Oct 30 02:18:24.701950 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.701804 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-socket\") pod \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " Oct 30 02:18:24.701950 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.701929 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-kube-api-access-m4wh7\" (UniqueName: \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-kube-api-access-m4wh7\") pod \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " Oct 30 02:18:24.702144 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.701997 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-envoy\") pod \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " Oct 30 02:18:24.702144 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.702039 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-podinfo\") pod \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " Oct 30 02:18:24.702144 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.702080 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-credential-socket\") pod \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " Oct 30 02:18:24.702319 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.702157 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-certs\") pod \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " Oct 30 02:18:24.702319 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.702268 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-token\" (UniqueName: \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-token\") pod \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\" (UID: \"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6\") " Oct 30 02:18:24.703353 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.703156 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-credential-socket" pod "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" (UID: "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:24.703697 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.703642 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istiod-ca-cert" pod "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" (UID: "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:24.703860 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.703831 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-certs" pod "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" (UID: "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:24.703993 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.703969 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-socket" pod "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" (UID: "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:24.704115 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.704100 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-data" pod "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" (UID: "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:24.705794 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.705758 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-kube-api-access-m4wh7" pod "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" (UID: "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6"). InnerVolumeSpecName "kube-api-access-m4wh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:24.706760 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.706718 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-token" pod "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" (UID: "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:24.706964 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.706937 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-podinfo" pod "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" (UID: "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:24.707087 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.707065 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-envoy" pod "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" (UID: "dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:24.804638 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.804592 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:24.804638 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.804634 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4wh7\" (UniqueName: \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-kube-api-access-m4wh7\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:24.804638 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.804649 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:24.804979 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.804663 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:24.804979 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.804677 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:24.804979 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.804690 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:24.804979 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.804706 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:24.804979 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.804719 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:24.804979 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:24.804731 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:25.178734 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:25.177845 5055 generic.go:356] "Generic (PLEG): container finished" podID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerID="b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe" exitCode=0 Oct 30 02:18:25.178734 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:25.177909 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" Oct 30 02:18:25.178734 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:25.177932 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" event={"ID":"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6","Type":"ContainerDied","Data":"b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe"} Oct 30 02:18:25.178734 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:25.177993 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx" event={"ID":"dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6","Type":"ContainerDied","Data":"b3e3bb7ed84fae8868840c73466d4bf77513ef0a97f7dc2a78ef9278ee7a78ef"} Oct 30 02:18:25.178734 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:25.178020 5055 scope.go:122] "RemoveContainer" containerID="b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe" Oct 30 02:18:25.198708 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:25.198679 5055 scope.go:122] "RemoveContainer" containerID="b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe" Oct 30 02:18:25.199120 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:25.199089 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe\": container with ID starting with b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe not found: ID does not exist" containerID="b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe" Oct 30 02:18:25.199237 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:25.199136 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe"} err="failed to get container status \"b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe\": rpc error: code = NotFound desc = could not find container \"b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe\": container with ID starting with b75766d0835de79c58f7c3317efe3b7e43b60a2aa8864ef28031caa8ba36d9fe not found: ID does not exist" Oct 30 02:18:25.230782 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:25.230745 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx"] Oct 30 02:18:25.243294 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:25.243254 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-54ghx"] Oct 30 02:18:26.701709 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.701678 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 02:18:26.835371 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.835295 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" path="/var/lib/kubelet/pods/dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6/volumes" Oct 30 02:18:26.841358 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841306 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-socket\") pod \"79831cb6-7cc2-4153-b501-079ef3bbe670\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " Oct 30 02:18:26.841540 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841413 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-token\" (UniqueName: \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-token\") pod \"79831cb6-7cc2-4153-b501-079ef3bbe670\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " Oct 30 02:18:26.841540 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841480 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-credential-socket\") pod \"79831cb6-7cc2-4153-b501-079ef3bbe670\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " Oct 30 02:18:26.841540 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841520 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-podinfo\") pod \"79831cb6-7cc2-4153-b501-079ef3bbe670\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " Oct 30 02:18:26.841741 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841550 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-envoy\") pod \"79831cb6-7cc2-4153-b501-079ef3bbe670\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " Oct 30 02:18:26.841741 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841603 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-socket" pod "79831cb6-7cc2-4153-b501-079ef3bbe670" (UID: "79831cb6-7cc2-4153-b501-079ef3bbe670"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:26.841741 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841649 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79831cb6-7cc2-4153-b501-079ef3bbe670-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79831cb6-7cc2-4153-b501-079ef3bbe670-istiod-ca-cert\") pod \"79831cb6-7cc2-4153-b501-079ef3bbe670\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " Oct 30 02:18:26.841912 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841764 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-kube-api-access-z6rzb\" (UniqueName: \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-kube-api-access-z6rzb\") pod \"79831cb6-7cc2-4153-b501-079ef3bbe670\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " Oct 30 02:18:26.841912 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841775 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-credential-socket" pod "79831cb6-7cc2-4153-b501-079ef3bbe670" (UID: "79831cb6-7cc2-4153-b501-079ef3bbe670"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:26.841912 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841803 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-data\") pod \"79831cb6-7cc2-4153-b501-079ef3bbe670\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " Oct 30 02:18:26.841912 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.841859 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-certs\") pod \"79831cb6-7cc2-4153-b501-079ef3bbe670\" (UID: \"79831cb6-7cc2-4153-b501-079ef3bbe670\") " Oct 30 02:18:26.842134 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.842041 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79831cb6-7cc2-4153-b501-079ef3bbe670-istiod-ca-cert" pod "79831cb6-7cc2-4153-b501-079ef3bbe670" (UID: "79831cb6-7cc2-4153-b501-079ef3bbe670"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:26.842234 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.842208 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-data" pod "79831cb6-7cc2-4153-b501-079ef3bbe670" (UID: "79831cb6-7cc2-4153-b501-079ef3bbe670"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:26.842304 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.842253 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-certs" pod "79831cb6-7cc2-4153-b501-079ef3bbe670" (UID: "79831cb6-7cc2-4153-b501-079ef3bbe670"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:26.842803 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.842733 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79831cb6-7cc2-4153-b501-079ef3bbe670-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:26.842803 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.842763 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:26.842803 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.842778 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:26.842803 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.842791 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:26.842803 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.842805 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:26.844106 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.844070 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-token" pod "79831cb6-7cc2-4153-b501-079ef3bbe670" (UID: "79831cb6-7cc2-4153-b501-079ef3bbe670"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:26.844596 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.844575 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-envoy" pod "79831cb6-7cc2-4153-b501-079ef3bbe670" (UID: "79831cb6-7cc2-4153-b501-079ef3bbe670"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:26.844686 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.844593 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-podinfo" pod "79831cb6-7cc2-4153-b501-079ef3bbe670" (UID: "79831cb6-7cc2-4153-b501-079ef3bbe670"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:26.844686 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.844599 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-kube-api-access-z6rzb" pod "79831cb6-7cc2-4153-b501-079ef3bbe670" (UID: "79831cb6-7cc2-4153-b501-079ef3bbe670"). InnerVolumeSpecName "kube-api-access-z6rzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:26.944113 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.944019 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6rzb\" (UniqueName: \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-kube-api-access-z6rzb\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:26.944113 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.944055 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:26.944113 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.944068 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:26.944113 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:26.944081 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79831cb6-7cc2-4153-b501-079ef3bbe670-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:27.196930 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.196835 5055 generic.go:356] "Generic (PLEG): container finished" podID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerID="5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1" exitCode=0 Oct 30 02:18:27.196930 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.196922 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" Oct 30 02:18:27.197186 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.196917 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" event={"ID":"79831cb6-7cc2-4153-b501-079ef3bbe670","Type":"ContainerDied","Data":"5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1"} Oct 30 02:18:27.197186 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.197060 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k" event={"ID":"79831cb6-7cc2-4153-b501-079ef3bbe670","Type":"ContainerDied","Data":"71fd595f51960577aa123a2ed184c54fbfaf2847f73005ff69fbfb9b2f116d3f"} Oct 30 02:18:27.197186 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.197090 5055 scope.go:122] "RemoveContainer" containerID="5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1" Oct 30 02:18:27.213259 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.211483 5055 scope.go:122] "RemoveContainer" containerID="d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549" Oct 30 02:18:27.224032 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.224000 5055 scope.go:122] "RemoveContainer" containerID="5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1" Oct 30 02:18:27.224378 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:27.224352 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1\": container with ID starting with 5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1 not found: ID does not exist" containerID="5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1" Oct 30 02:18:27.224449 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.224391 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1"} err="failed to get container status \"5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1\": rpc error: code = NotFound desc = could not find container \"5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1\": container with ID starting with 5e05bc13ae149504a103380e79d61305442a199eb03be20e93f34db1dae96bc1 not found: ID does not exist" Oct 30 02:18:27.224449 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.224410 5055 scope.go:122] "RemoveContainer" containerID="d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549" Oct 30 02:18:27.224724 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:27.224708 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549\": container with ID starting with d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549 not found: ID does not exist" containerID="d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549" Oct 30 02:18:27.224793 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.224726 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549"} err="failed to get container status \"d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549\": rpc error: code = NotFound desc = could not find container \"d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549\": container with ID starting with d6a23db1c44282f214d0f85dd40340dffec240f96b9f218ffa0b81339b2f0549 not found: ID does not exist" Oct 30 02:18:27.263516 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.263477 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k"] Oct 30 02:18:27.275928 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:27.275888 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-76a19e-openshift-default-59d8764697-w477k"] Oct 30 02:18:28.821170 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:28.821138 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" path="/var/lib/kubelet/pods/79831cb6-7cc2-4153-b501-079ef3bbe670/volumes" Oct 30 02:18:29.950452 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:29.950412 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" podUID="65f34b67-7d82-405f-8825-8fede2564631" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:29.950875 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:29.950502 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" podUID="1ae14669-6647-4386-a4fd-7a769705984e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:31.425846 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.425817 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 02:18:31.473727 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.473701 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 02:18:31.499740 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.499707 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ae14669-6647-4386-a4fd-7a769705984e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae14669-6647-4386-a4fd-7a769705984e-plugins-config-volume\") pod \"1ae14669-6647-4386-a4fd-7a769705984e\" (UID: \"1ae14669-6647-4386-a4fd-7a769705984e\") " Oct 30 02:18:31.499977 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.499805 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65f34b67-7d82-405f-8825-8fede2564631-kube-api-access-f8cmj\" (UniqueName: \"kubernetes.io/projected/65f34b67-7d82-405f-8825-8fede2564631-kube-api-access-f8cmj\") pod \"65f34b67-7d82-405f-8825-8fede2564631\" (UID: \"65f34b67-7d82-405f-8825-8fede2564631\") " Oct 30 02:18:31.499977 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.499914 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae14669-6647-4386-a4fd-7a769705984e-kube-api-access-wfsrn\" (UniqueName: \"kubernetes.io/projected/1ae14669-6647-4386-a4fd-7a769705984e-kube-api-access-wfsrn\") pod \"1ae14669-6647-4386-a4fd-7a769705984e\" (UID: \"1ae14669-6647-4386-a4fd-7a769705984e\") " Oct 30 02:18:31.500118 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.500089 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65f34b67-7d82-405f-8825-8fede2564631-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f34b67-7d82-405f-8825-8fede2564631-plugins-config-volume\") pod \"65f34b67-7d82-405f-8825-8fede2564631\" (UID: \"65f34b67-7d82-405f-8825-8fede2564631\") " Oct 30 02:18:31.500231 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.500202 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ae14669-6647-4386-a4fd-7a769705984e-plugins-config-volume" pod "1ae14669-6647-4386-a4fd-7a769705984e" (UID: "1ae14669-6647-4386-a4fd-7a769705984e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:31.500535 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.500517 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ae14669-6647-4386-a4fd-7a769705984e-plugins-config-volume\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.500697 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.500670 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f34b67-7d82-405f-8825-8fede2564631-plugins-config-volume" pod "65f34b67-7d82-405f-8825-8fede2564631" (UID: "65f34b67-7d82-405f-8825-8fede2564631"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:31.503203 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.503172 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f34b67-7d82-405f-8825-8fede2564631-kube-api-access-f8cmj" pod "65f34b67-7d82-405f-8825-8fede2564631" (UID: "65f34b67-7d82-405f-8825-8fede2564631"). InnerVolumeSpecName "kube-api-access-f8cmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:31.503311 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.503218 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae14669-6647-4386-a4fd-7a769705984e-kube-api-access-wfsrn" pod "1ae14669-6647-4386-a4fd-7a769705984e" (UID: "1ae14669-6647-4386-a4fd-7a769705984e"). InnerVolumeSpecName "kube-api-access-wfsrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:31.601525 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.601475 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8cmj\" (UniqueName: \"kubernetes.io/projected/65f34b67-7d82-405f-8825-8fede2564631-kube-api-access-f8cmj\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.601525 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.601523 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfsrn\" (UniqueName: \"kubernetes.io/projected/1ae14669-6647-4386-a4fd-7a769705984e-kube-api-access-wfsrn\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:31.601768 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:31.601538 5055 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f34b67-7d82-405f-8825-8fede2564631-plugins-config-volume\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:32.236108 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.236063 5055 generic.go:356] "Generic (PLEG): container finished" podID="65f34b67-7d82-405f-8825-8fede2564631" containerID="ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99" exitCode=0 Oct 30 02:18:32.236372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.236139 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" Oct 30 02:18:32.236372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.236150 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" event={"ID":"65f34b67-7d82-405f-8825-8fede2564631","Type":"ContainerDied","Data":"ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99"} Oct 30 02:18:32.236372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.236230 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk" event={"ID":"65f34b67-7d82-405f-8825-8fede2564631","Type":"ContainerDied","Data":"dc2655e893591d86f5c395f3bca42ca5579acd136b80bb2814f4c2819b10fc96"} Oct 30 02:18:32.236372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.236260 5055 scope.go:122] "RemoveContainer" containerID="ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99" Oct 30 02:18:32.241624 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.241597 5055 generic.go:356] "Generic (PLEG): container finished" podID="1ae14669-6647-4386-a4fd-7a769705984e" containerID="061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8" exitCode=0 Oct 30 02:18:32.241786 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.241673 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" Oct 30 02:18:32.241786 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.241679 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" event={"ID":"1ae14669-6647-4386-a4fd-7a769705984e","Type":"ContainerDied","Data":"061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8"} Oct 30 02:18:32.241786 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.241744 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw" event={"ID":"1ae14669-6647-4386-a4fd-7a769705984e","Type":"ContainerDied","Data":"64d86e0801379142db61911ed1d207bc94b2db83e7b53b495ada74eff2f2ac76"} Oct 30 02:18:32.247891 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.247869 5055 scope.go:122] "RemoveContainer" containerID="ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99" Oct 30 02:18:32.248218 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:32.248190 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99\": container with ID starting with ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99 not found: ID does not exist" containerID="ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99" Oct 30 02:18:32.248277 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.248233 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99"} err="failed to get container status \"ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99\": rpc error: code = NotFound desc = could not find container \"ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99\": container with ID starting with ecfbf33d29b41be268a3e615b1dcbad7dec71464293fd3639064b2693304fe99 not found: ID does not exist" Oct 30 02:18:32.248277 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.248259 5055 scope.go:122] "RemoveContainer" containerID="061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8" Oct 30 02:18:32.258945 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.258759 5055 scope.go:122] "RemoveContainer" containerID="061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8" Oct 30 02:18:32.259064 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:32.259044 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8\": container with ID starting with 061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8 not found: ID does not exist" containerID="061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8" Oct 30 02:18:32.259121 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.259073 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8"} err="failed to get container status \"061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8\": rpc error: code = NotFound desc = could not find container \"061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8\": container with ID starting with 061edf06951c8e3499f1cb00b26cc9c8caf544c76a16890bcf35227abb3203e8 not found: ID does not exist" Oct 30 02:18:32.303523 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.303479 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk"] Oct 30 02:18:32.315881 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.315846 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-76a19e-5c4559bf9d-dj9jk"] Oct 30 02:18:32.345076 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.345035 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw"] Oct 30 02:18:32.352797 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.352762 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-76a19e-75c45bdc-4cbtw"] Oct 30 02:18:32.821533 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.821493 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae14669-6647-4386-a4fd-7a769705984e" path="/var/lib/kubelet/pods/1ae14669-6647-4386-a4fd-7a769705984e/volumes" Oct 30 02:18:32.821970 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:32.821955 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65f34b67-7d82-405f-8825-8fede2564631" path="/var/lib/kubelet/pods/65f34b67-7d82-405f-8825-8fede2564631/volumes" Oct 30 02:18:38.084871 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:38.084838 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv"] Oct 30 02:18:38.085450 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:38.085117 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" containerID="cri-o://2c2c0a94b39cca4e37a4a11498a1aed0b502c1b7f503a2e2450084fc15a16a6c" gracePeriod=30 Oct 30 02:18:41.748123 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:41.748081 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8"] Oct 30 02:18:41.748667 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:41.748356 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" podUID="64ecaa40-4720-4849-8d03-892807164526" containerName="maas-api" containerID="cri-o://156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7" gracePeriod=30 Oct 30 02:18:42.196165 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.196133 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 02:18:42.311730 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.311694 5055 generic.go:356] "Generic (PLEG): container finished" podID="64ecaa40-4720-4849-8d03-892807164526" containerID="156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7" exitCode=0 Oct 30 02:18:42.311936 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.311747 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" Oct 30 02:18:42.311936 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.311780 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" event={"ID":"64ecaa40-4720-4849-8d03-892807164526","Type":"ContainerDied","Data":"156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7"} Oct 30 02:18:42.311936 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.311833 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" event={"ID":"64ecaa40-4720-4849-8d03-892807164526","Type":"ContainerDied","Data":"81ee7a7eb4cf2345ebef64a769a3e5d65339c0e795cc9d09cec177afb9358c79"} Oct 30 02:18:42.311936 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.311855 5055 scope.go:122] "RemoveContainer" containerID="156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7" Oct 30 02:18:42.322527 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.322506 5055 scope.go:122] "RemoveContainer" containerID="156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7" Oct 30 02:18:42.322891 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:42.322867 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7\": container with ID starting with 156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7 not found: ID does not exist" containerID="156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7" Oct 30 02:18:42.323004 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.322901 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7"} err="failed to get container status \"156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7\": rpc error: code = NotFound desc = could not find container \"156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7\": container with ID starting with 156452884cc6918567cb6539c453a9373350c95a2571078dfe7a4c84d770b5c7 not found: ID does not exist" Oct 30 02:18:42.349674 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.349587 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64ecaa40-4720-4849-8d03-892807164526-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ecaa40-4720-4849-8d03-892807164526-openshift-service-ca\") pod \"64ecaa40-4720-4849-8d03-892807164526\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " Oct 30 02:18:42.349674 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.349664 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-maas-api-tls\") pod \"64ecaa40-4720-4849-8d03-892807164526\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " Oct 30 02:18:42.349925 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.349781 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-metrics-certs\") pod \"64ecaa40-4720-4849-8d03-892807164526\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " Oct 30 02:18:42.350153 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.349999 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64ecaa40-4720-4849-8d03-892807164526-kube-api-access-8f57n\" (UniqueName: \"kubernetes.io/projected/64ecaa40-4720-4849-8d03-892807164526-kube-api-access-8f57n\") pod \"64ecaa40-4720-4849-8d03-892807164526\" (UID: \"64ecaa40-4720-4849-8d03-892807164526\") " Oct 30 02:18:42.350153 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.350056 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64ecaa40-4720-4849-8d03-892807164526-openshift-service-ca" pod "64ecaa40-4720-4849-8d03-892807164526" (UID: "64ecaa40-4720-4849-8d03-892807164526"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:42.350996 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.350911 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64ecaa40-4720-4849-8d03-892807164526-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:42.352682 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.352642 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-maas-api-tls" pod "64ecaa40-4720-4849-8d03-892807164526" (UID: "64ecaa40-4720-4849-8d03-892807164526"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:42.353717 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.353697 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-metrics-certs" pod "64ecaa40-4720-4849-8d03-892807164526" (UID: "64ecaa40-4720-4849-8d03-892807164526"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:42.354515 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.354490 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64ecaa40-4720-4849-8d03-892807164526-kube-api-access-8f57n" pod "64ecaa40-4720-4849-8d03-892807164526" (UID: "64ecaa40-4720-4849-8d03-892807164526"). InnerVolumeSpecName "kube-api-access-8f57n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:42.452285 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.452197 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f57n\" (UniqueName: \"kubernetes.io/projected/64ecaa40-4720-4849-8d03-892807164526-kube-api-access-8f57n\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:42.452285 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.452250 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:42.452285 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.452262 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ecaa40-4720-4849-8d03-892807164526-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:42.661034 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.660992 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8"] Oct 30 02:18:42.665420 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.665387 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8"] Oct 30 02:18:42.823152 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:42.822859 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64ecaa40-4720-4849-8d03-892807164526" path="/var/lib/kubelet/pods/64ecaa40-4720-4849-8d03-892807164526/volumes" Oct 30 02:18:43.324394 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.324317 5055 generic.go:356] "Generic (PLEG): container finished" podID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerID="2c2c0a94b39cca4e37a4a11498a1aed0b502c1b7f503a2e2450084fc15a16a6c" exitCode=0 Oct 30 02:18:43.324596 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.324389 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" event={"ID":"34e7b35c-1ce6-4b22-9c80-f6745b3727b3","Type":"ContainerDied","Data":"2c2c0a94b39cca4e37a4a11498a1aed0b502c1b7f503a2e2450084fc15a16a6c"} Oct 30 02:18:43.383236 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.383207 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 02:18:43.464692 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.464653 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-credential-socket\") pod \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " Oct 30 02:18:43.464886 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.464751 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-socket\") pod \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " Oct 30 02:18:43.464886 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.464836 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-credential-socket" pod "34e7b35c-1ce6-4b22-9c80-f6745b3727b3" (UID: "34e7b35c-1ce6-4b22-9c80-f6745b3727b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:43.464886 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.464849 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-envoy\") pod \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " Oct 30 02:18:43.465073 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.464906 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-certs\") pod \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " Oct 30 02:18:43.465073 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.464931 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-data\") pod \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " Oct 30 02:18:43.465073 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465014 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-kube-api-access-6c4bj\" (UniqueName: \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-kube-api-access-6c4bj\") pod \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " Oct 30 02:18:43.465073 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465060 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istiod-ca-cert\") pod \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " Oct 30 02:18:43.465293 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465099 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-token\" (UniqueName: \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-token\") pod \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " Oct 30 02:18:43.465293 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465096 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-socket" pod "34e7b35c-1ce6-4b22-9c80-f6745b3727b3" (UID: "34e7b35c-1ce6-4b22-9c80-f6745b3727b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:43.465293 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465132 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-podinfo\") pod \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\" (UID: \"34e7b35c-1ce6-4b22-9c80-f6745b3727b3\") " Oct 30 02:18:43.465492 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465450 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istiod-ca-cert" pod "34e7b35c-1ce6-4b22-9c80-f6745b3727b3" (UID: "34e7b35c-1ce6-4b22-9c80-f6745b3727b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:43.465574 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465550 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-data" pod "34e7b35c-1ce6-4b22-9c80-f6745b3727b3" (UID: "34e7b35c-1ce6-4b22-9c80-f6745b3727b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:43.465776 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465756 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:43.465776 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465778 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:43.466057 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465791 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:43.466057 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.465804 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:43.466197 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.466075 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-certs" pod "34e7b35c-1ce6-4b22-9c80-f6745b3727b3" (UID: "34e7b35c-1ce6-4b22-9c80-f6745b3727b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:43.467983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.467955 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-envoy" pod "34e7b35c-1ce6-4b22-9c80-f6745b3727b3" (UID: "34e7b35c-1ce6-4b22-9c80-f6745b3727b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:43.468128 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.468104 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-token" pod "34e7b35c-1ce6-4b22-9c80-f6745b3727b3" (UID: "34e7b35c-1ce6-4b22-9c80-f6745b3727b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:43.468187 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.468143 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-kube-api-access-6c4bj" pod "34e7b35c-1ce6-4b22-9c80-f6745b3727b3" (UID: "34e7b35c-1ce6-4b22-9c80-f6745b3727b3"). InnerVolumeSpecName "kube-api-access-6c4bj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:43.468923 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.468897 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-podinfo" pod "34e7b35c-1ce6-4b22-9c80-f6745b3727b3" (UID: "34e7b35c-1ce6-4b22-9c80-f6745b3727b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:43.567260 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.567219 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:43.567260 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.567260 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:43.567486 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.567275 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:43.567486 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.567288 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:43.567486 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.567302 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c4bj\" (UniqueName: \"kubernetes.io/projected/34e7b35c-1ce6-4b22-9c80-f6745b3727b3-kube-api-access-6c4bj\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:43.804584 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.804549 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk"] Oct 30 02:18:43.805081 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:43.804796 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" podUID="6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2" containerName="maas-api" containerID="cri-o://8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4" gracePeriod=30 Oct 30 02:18:44.192357 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.187529 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 02:18:44.292744 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.292706 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-maas-api-tls\") pod \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " Oct 30 02:18:44.292949 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.292844 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-openshift-service-ca\") pod \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " Oct 30 02:18:44.292949 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.292887 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-kube-api-access-8nssc\" (UniqueName: \"kubernetes.io/projected/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-kube-api-access-8nssc\") pod \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " Oct 30 02:18:44.292949 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.292910 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-metrics-certs\") pod \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\" (UID: \"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2\") " Oct 30 02:18:44.293228 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.293198 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-openshift-service-ca" pod "6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2" (UID: "6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:44.295287 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.295264 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-metrics-certs" pod "6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2" (UID: "6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:44.295675 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.295645 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-kube-api-access-8nssc" pod "6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2" (UID: "6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2"). InnerVolumeSpecName "kube-api-access-8nssc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:44.295755 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.295714 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-maas-api-tls" pod "6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2" (UID: "6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:44.334797 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.334771 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" Oct 30 02:18:44.335002 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.334796 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv" event={"ID":"34e7b35c-1ce6-4b22-9c80-f6745b3727b3","Type":"ContainerDied","Data":"f082b27fe8293c6ff7abc5168c3dd5f595e49b1517664444d196b318f3d48174"} Oct 30 02:18:44.335002 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.334863 5055 scope.go:122] "RemoveContainer" containerID="2c2c0a94b39cca4e37a4a11498a1aed0b502c1b7f503a2e2450084fc15a16a6c" Oct 30 02:18:44.336600 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.336575 5055 generic.go:356] "Generic (PLEG): container finished" podID="6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2" containerID="8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4" exitCode=0 Oct 30 02:18:44.336835 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.336657 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" event={"ID":"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2","Type":"ContainerDied","Data":"8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4"} Oct 30 02:18:44.336835 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.336687 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" event={"ID":"6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2","Type":"ContainerDied","Data":"70dd6e8fc5736275d36aaa55e5071a8eabcfdd52f89aa39f04c0e8b4d6cb18f3"} Oct 30 02:18:44.336835 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.336755 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk" Oct 30 02:18:44.350606 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.350586 5055 scope.go:122] "RemoveContainer" containerID="8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4" Oct 30 02:18:44.361644 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.361617 5055 scope.go:122] "RemoveContainer" containerID="8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4" Oct 30 02:18:44.361968 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:44.361940 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4\": container with ID starting with 8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4 not found: ID does not exist" containerID="8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4" Oct 30 02:18:44.362043 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.361980 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4"} err="failed to get container status \"8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4\": rpc error: code = NotFound desc = could not find container \"8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4\": container with ID starting with 8aee8a0010a05f0d1cc4a04de258f77cbcd736daac2ada48ea69fc183a6e5bb4 not found: ID does not exist" Oct 30 02:18:44.389227 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.389150 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk"] Oct 30 02:18:44.393175 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.393144 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-5fb950-6f86b5b8b4-ztzvk"] Oct 30 02:18:44.394800 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.394771 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:44.394800 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.394798 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nssc\" (UniqueName: \"kubernetes.io/projected/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-kube-api-access-8nssc\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:44.394953 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.394810 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:44.394953 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.394821 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:44.429744 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.429708 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv"] Oct 30 02:18:44.438422 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.438391 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-jhpgv"] Oct 30 02:18:44.819550 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.819515 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" path="/var/lib/kubelet/pods/34e7b35c-1ce6-4b22-9c80-f6745b3727b3/volumes" Oct 30 02:18:44.819945 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:44.819931 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2" path="/var/lib/kubelet/pods/6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2/volumes" Oct 30 02:18:45.028487 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:45.028436 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-2e618f-77cdb98d8f-t77p8" podUID="64ecaa40-4720-4849-8d03-892807164526" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.167:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:18:49.220002 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:49.219967 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:18:49.547638 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:49.547556 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt"] Oct 30 02:18:49.548015 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:49.547987 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerName="istio-proxy" containerID="cri-o://957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944" gracePeriod=30 Oct 30 02:18:51.160135 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:51.160100 5055 patch_prober.go:29] interesting pod/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:18:51.160677 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:51.160158 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:18:54.843347 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.843310 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 02:18:54.922722 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.922666 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-kube-api-access-4xmnv\" (UniqueName: \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-kube-api-access-4xmnv\") pod \"c6e20bcb-70f4-416a-a159-73e28ad6e931\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " Oct 30 02:18:54.922946 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.922749 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-certs\") pod \"c6e20bcb-70f4-416a-a159-73e28ad6e931\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " Oct 30 02:18:54.922946 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.922795 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-credential-socket\") pod \"c6e20bcb-70f4-416a-a159-73e28ad6e931\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " Oct 30 02:18:54.922946 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.922849 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-socket\") pod \"c6e20bcb-70f4-416a-a159-73e28ad6e931\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " Oct 30 02:18:54.922946 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.922882 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-data\") pod \"c6e20bcb-70f4-416a-a159-73e28ad6e931\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " Oct 30 02:18:54.922946 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.922915 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6e20bcb-70f4-416a-a159-73e28ad6e931-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e20bcb-70f4-416a-a159-73e28ad6e931-istiod-ca-cert\") pod \"c6e20bcb-70f4-416a-a159-73e28ad6e931\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " Oct 30 02:18:54.923221 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.923014 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-podinfo\") pod \"c6e20bcb-70f4-416a-a159-73e28ad6e931\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " Oct 30 02:18:54.923221 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.923106 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-envoy\") pod \"c6e20bcb-70f4-416a-a159-73e28ad6e931\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " Oct 30 02:18:54.923221 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.923159 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-token\" (UniqueName: \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-token\") pod \"c6e20bcb-70f4-416a-a159-73e28ad6e931\" (UID: \"c6e20bcb-70f4-416a-a159-73e28ad6e931\") " Oct 30 02:18:54.926904 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.926870 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-token" pod "c6e20bcb-70f4-416a-a159-73e28ad6e931" (UID: "c6e20bcb-70f4-416a-a159-73e28ad6e931"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:54.927748 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.927714 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-data" pod "c6e20bcb-70f4-416a-a159-73e28ad6e931" (UID: "c6e20bcb-70f4-416a-a159-73e28ad6e931"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:54.928189 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.928139 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-certs" pod "c6e20bcb-70f4-416a-a159-73e28ad6e931" (UID: "c6e20bcb-70f4-416a-a159-73e28ad6e931"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:54.928400 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.928339 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-credential-socket" pod "c6e20bcb-70f4-416a-a159-73e28ad6e931" (UID: "c6e20bcb-70f4-416a-a159-73e28ad6e931"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:54.929155 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.928768 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-socket" pod "c6e20bcb-70f4-416a-a159-73e28ad6e931" (UID: "c6e20bcb-70f4-416a-a159-73e28ad6e931"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:54.929315 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.929144 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e20bcb-70f4-416a-a159-73e28ad6e931-istiod-ca-cert" pod "c6e20bcb-70f4-416a-a159-73e28ad6e931" (UID: "c6e20bcb-70f4-416a-a159-73e28ad6e931"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:54.931379 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.931312 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-kube-api-access-4xmnv" pod "c6e20bcb-70f4-416a-a159-73e28ad6e931" (UID: "c6e20bcb-70f4-416a-a159-73e28ad6e931"). InnerVolumeSpecName "kube-api-access-4xmnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:54.931971 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.931940 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-podinfo" pod "c6e20bcb-70f4-416a-a159-73e28ad6e931" (UID: "c6e20bcb-70f4-416a-a159-73e28ad6e931"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:54.937503 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:54.937481 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-envoy" pod "c6e20bcb-70f4-416a-a159-73e28ad6e931" (UID: "c6e20bcb-70f4-416a-a159-73e28ad6e931"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:55.028258 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.025940 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.028258 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.025995 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmnv\" (UniqueName: \"kubernetes.io/projected/c6e20bcb-70f4-416a-a159-73e28ad6e931-kube-api-access-4xmnv\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.028258 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.026011 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.028258 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.026025 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.028258 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.026047 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.028258 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.026060 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.028258 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.026074 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e20bcb-70f4-416a-a159-73e28ad6e931-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.028258 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.026085 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.028258 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.026102 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e20bcb-70f4-416a-a159-73e28ad6e931-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:18:55.428387 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.428351 5055 generic.go:356] "Generic (PLEG): container finished" podID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerID="957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944" exitCode=0 Oct 30 02:18:55.428571 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.428414 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" Oct 30 02:18:55.428571 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.428418 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" event={"ID":"c6e20bcb-70f4-416a-a159-73e28ad6e931","Type":"ContainerDied","Data":"957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944"} Oct 30 02:18:55.428571 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.428510 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt" event={"ID":"c6e20bcb-70f4-416a-a159-73e28ad6e931","Type":"ContainerDied","Data":"8d5bfeb09b4f1ae19105a3ed4d95ae6961eca294fd67ecf32442b6444da38847"} Oct 30 02:18:55.428571 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.428534 5055 scope.go:122] "RemoveContainer" containerID="957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944" Oct 30 02:18:55.439094 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.439071 5055 scope.go:122] "RemoveContainer" containerID="57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183" Oct 30 02:18:55.448614 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.448590 5055 scope.go:122] "RemoveContainer" containerID="957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944" Oct 30 02:18:55.448970 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:55.448946 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944\": container with ID starting with 957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944 not found: ID does not exist" containerID="957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944" Oct 30 02:18:55.449091 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.448977 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944"} err="failed to get container status \"957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944\": rpc error: code = NotFound desc = could not find container \"957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944\": container with ID starting with 957bfea1bb86aacb1666b440a9eb579ae545bee8182f1b2d9b3d0b9d932bf944 not found: ID does not exist" Oct 30 02:18:55.449091 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.449000 5055 scope.go:122] "RemoveContainer" containerID="57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183" Oct 30 02:18:55.449262 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:18:55.449244 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183\": container with ID starting with 57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183 not found: ID does not exist" containerID="57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183" Oct 30 02:18:55.449309 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.449269 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183"} err="failed to get container status \"57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183\": rpc error: code = NotFound desc = could not find container \"57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183\": container with ID starting with 57a1b4511d63dff73fa09209a3b0c939e75ed1206d18f3df87be52554ba48183 not found: ID does not exist" Oct 30 02:18:55.495492 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.495458 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt"] Oct 30 02:18:55.507786 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:55.507750 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2e618f-openshift-default-565df787c-9pbrt"] Oct 30 02:18:56.819741 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:18:56.819706 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" path="/var/lib/kubelet/pods/c6e20bcb-70f4-416a-a159-73e28ad6e931/volumes" Oct 30 02:19:14.598761 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:14.598699 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd"] Oct 30 02:19:14.599306 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:14.599002 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" containerID="cri-o://775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f" gracePeriod=30 Oct 30 02:19:19.221039 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:19:19.220999 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:19:19.901404 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.901378 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 02:19:19.953505 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.953466 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-token\" (UniqueName: \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-token\") pod \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " Oct 30 02:19:19.953766 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.953535 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-kube-api-access-j55fp\" (UniqueName: \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-kube-api-access-j55fp\") pod \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " Oct 30 02:19:19.953766 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.953611 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-certs\") pod \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " Oct 30 02:19:19.953766 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.953686 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istiod-ca-cert\") pod \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " Oct 30 02:19:19.953766 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.953736 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-data\") pod \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " Oct 30 02:19:19.954010 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.953777 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-credential-socket\") pod \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " Oct 30 02:19:19.954010 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.953865 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-envoy\") pod \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " Oct 30 02:19:19.954010 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.953949 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-podinfo\") pod \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " Oct 30 02:19:19.954010 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.953977 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-socket\") pod \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\" (UID: \"d58c8f32-415f-40bd-ad3f-275b80c34f9a\") " Oct 30 02:19:19.954248 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.954208 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-certs" pod "d58c8f32-415f-40bd-ad3f-275b80c34f9a" (UID: "d58c8f32-415f-40bd-ad3f-275b80c34f9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:19.954397 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.954362 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-data" pod "d58c8f32-415f-40bd-ad3f-275b80c34f9a" (UID: "d58c8f32-415f-40bd-ad3f-275b80c34f9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:19.954473 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.954390 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istiod-ca-cert" pod "d58c8f32-415f-40bd-ad3f-275b80c34f9a" (UID: "d58c8f32-415f-40bd-ad3f-275b80c34f9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:19.954588 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.954562 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-socket" pod "d58c8f32-415f-40bd-ad3f-275b80c34f9a" (UID: "d58c8f32-415f-40bd-ad3f-275b80c34f9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:19.954814 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.954792 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:19.954931 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.954820 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:19.954931 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.954834 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:19.954931 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.954846 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:19.955934 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.955907 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-credential-socket" pod "d58c8f32-415f-40bd-ad3f-275b80c34f9a" (UID: "d58c8f32-415f-40bd-ad3f-275b80c34f9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:19.956913 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.956877 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-token" pod "d58c8f32-415f-40bd-ad3f-275b80c34f9a" (UID: "d58c8f32-415f-40bd-ad3f-275b80c34f9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:19.956913 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.956902 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-kube-api-access-j55fp" pod "d58c8f32-415f-40bd-ad3f-275b80c34f9a" (UID: "d58c8f32-415f-40bd-ad3f-275b80c34f9a"). InnerVolumeSpecName "kube-api-access-j55fp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:19.957251 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.957227 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-podinfo" pod "d58c8f32-415f-40bd-ad3f-275b80c34f9a" (UID: "d58c8f32-415f-40bd-ad3f-275b80c34f9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:19:19.957941 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:19.957916 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-envoy" pod "d58c8f32-415f-40bd-ad3f-275b80c34f9a" (UID: "d58c8f32-415f-40bd-ad3f-275b80c34f9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:20.056253 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.056218 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.056253 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.056249 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.056253 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.056260 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j55fp\" (UniqueName: \"kubernetes.io/projected/d58c8f32-415f-40bd-ad3f-275b80c34f9a-kube-api-access-j55fp\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.056587 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.056269 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.056587 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.056278 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d58c8f32-415f-40bd-ad3f-275b80c34f9a-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:20.090365 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.090312 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw"] Oct 30 02:19:20.590947 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.590915 5055 generic.go:356] "Generic (PLEG): container finished" podID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerID="775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f" exitCode=0 Oct 30 02:19:20.590947 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.590959 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" Oct 30 02:19:20.591466 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.590994 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" event={"ID":"d58c8f32-415f-40bd-ad3f-275b80c34f9a","Type":"ContainerDied","Data":"775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f"} Oct 30 02:19:20.591466 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.591047 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd" event={"ID":"d58c8f32-415f-40bd-ad3f-275b80c34f9a","Type":"ContainerDied","Data":"b917a61a79de3a1515e0c53ee0fe967ce9e028c85821e9db4cf7f234a72c03c6"} Oct 30 02:19:20.591466 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.591069 5055 scope.go:122] "RemoveContainer" containerID="775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f" Oct 30 02:19:20.602454 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.602429 5055 scope.go:122] "RemoveContainer" containerID="238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d" Oct 30 02:19:20.611739 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.611714 5055 scope.go:122] "RemoveContainer" containerID="775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f" Oct 30 02:19:20.612087 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:19:20.612064 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f\": container with ID starting with 775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f not found: ID does not exist" containerID="775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f" Oct 30 02:19:20.612178 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.612098 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f"} err="failed to get container status \"775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f\": rpc error: code = NotFound desc = could not find container \"775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f\": container with ID starting with 775877c78ab91e5e6a3890ffd33e5e58d235e7e5cd243c03220e60217b799a1f not found: ID does not exist" Oct 30 02:19:20.612178 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.612116 5055 scope.go:122] "RemoveContainer" containerID="238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d" Oct 30 02:19:20.612438 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:19:20.612350 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d\": container with ID starting with 238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d not found: ID does not exist" containerID="238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d" Oct 30 02:19:20.612438 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.612380 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d"} err="failed to get container status \"238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d\": rpc error: code = NotFound desc = could not find container \"238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d\": container with ID starting with 238f416c68ef1cae508dd90f76c503c4e66880cef0bb5608a908fb9daf8edb6d not found: ID does not exist" Oct 30 02:19:20.643928 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.643895 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd"] Oct 30 02:19:20.655925 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.655896 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4f7628-openshift-default-7d4cccf989-9jjfd"] Oct 30 02:19:20.820695 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:20.820661 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" path="/var/lib/kubelet/pods/d58c8f32-415f-40bd-ad3f-275b80c34f9a/volumes" Oct 30 02:19:30.802532 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:30.802443 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7"] Oct 30 02:19:30.803126 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:30.802700 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" podUID="b6f07429-1516-4228-85f1-1c37e3100ad7" containerName="maas-api" containerID="cri-o://2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a" gracePeriod=30 Oct 30 02:19:31.127487 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.127451 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 02:19:31.154977 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.151407 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx"] Oct 30 02:19:31.154977 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.151684 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" podUID="3337f877-9bf8-4787-b840-3a07f57c330f" containerName="maas-api" containerID="cri-o://be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e" gracePeriod=30 Oct 30 02:19:31.296049 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.295999 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f07429-1516-4228-85f1-1c37e3100ad7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f07429-1516-4228-85f1-1c37e3100ad7-openshift-service-ca\") pod \"b6f07429-1516-4228-85f1-1c37e3100ad7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " Oct 30 02:19:31.296261 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.296118 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-maas-api-tls\") pod \"b6f07429-1516-4228-85f1-1c37e3100ad7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " Oct 30 02:19:31.296261 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.296237 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-metrics-certs\") pod \"b6f07429-1516-4228-85f1-1c37e3100ad7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " Oct 30 02:19:31.296435 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.296271 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f07429-1516-4228-85f1-1c37e3100ad7-kube-api-access-88qsx\" (UniqueName: \"kubernetes.io/projected/b6f07429-1516-4228-85f1-1c37e3100ad7-kube-api-access-88qsx\") pod \"b6f07429-1516-4228-85f1-1c37e3100ad7\" (UID: \"b6f07429-1516-4228-85f1-1c37e3100ad7\") " Oct 30 02:19:31.296493 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.296456 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f07429-1516-4228-85f1-1c37e3100ad7-openshift-service-ca" pod "b6f07429-1516-4228-85f1-1c37e3100ad7" (UID: "b6f07429-1516-4228-85f1-1c37e3100ad7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:31.296790 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.296770 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f07429-1516-4228-85f1-1c37e3100ad7-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:31.299277 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.299240 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-maas-api-tls" pod "b6f07429-1516-4228-85f1-1c37e3100ad7" (UID: "b6f07429-1516-4228-85f1-1c37e3100ad7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:31.299463 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.299445 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-metrics-certs" pod "b6f07429-1516-4228-85f1-1c37e3100ad7" (UID: "b6f07429-1516-4228-85f1-1c37e3100ad7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:31.299583 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.299566 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f07429-1516-4228-85f1-1c37e3100ad7-kube-api-access-88qsx" pod "b6f07429-1516-4228-85f1-1c37e3100ad7" (UID: "b6f07429-1516-4228-85f1-1c37e3100ad7"). InnerVolumeSpecName "kube-api-access-88qsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:31.398224 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.398145 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:31.398224 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.398180 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b6f07429-1516-4228-85f1-1c37e3100ad7-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:31.398224 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.398190 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qsx\" (UniqueName: \"kubernetes.io/projected/b6f07429-1516-4228-85f1-1c37e3100ad7-kube-api-access-88qsx\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:31.656808 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.656745 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:19:31.659900 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.659879 5055 generic.go:356] "Generic (PLEG): container finished" podID="b6f07429-1516-4228-85f1-1c37e3100ad7" containerID="2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a" exitCode=0 Oct 30 02:19:31.660023 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.659917 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" Oct 30 02:19:31.660023 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.659958 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" event={"ID":"b6f07429-1516-4228-85f1-1c37e3100ad7","Type":"ContainerDied","Data":"2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a"} Oct 30 02:19:31.660023 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.660005 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7" event={"ID":"b6f07429-1516-4228-85f1-1c37e3100ad7","Type":"ContainerDied","Data":"d7ddb73f21855145fb37c477b140503afb8e2519be053f3d5abb59c3c89b949d"} Oct 30 02:19:31.660167 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.660032 5055 scope.go:122] "RemoveContainer" containerID="2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a" Oct 30 02:19:31.661302 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.661281 5055 generic.go:356] "Generic (PLEG): container finished" podID="3337f877-9bf8-4787-b840-3a07f57c330f" containerID="be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e" exitCode=0 Oct 30 02:19:31.661435 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.661308 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" event={"ID":"3337f877-9bf8-4787-b840-3a07f57c330f","Type":"ContainerDied","Data":"be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e"} Oct 30 02:19:31.661435 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.661354 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" Oct 30 02:19:31.661435 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.661377 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx" event={"ID":"3337f877-9bf8-4787-b840-3a07f57c330f","Type":"ContainerDied","Data":"19aae18c4acd10bdd980f6f6c2061bc64875d28837be11ad3423de93deaa26e9"} Oct 30 02:19:31.673250 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.671533 5055 scope.go:122] "RemoveContainer" containerID="2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a" Oct 30 02:19:31.673250 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:19:31.672022 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a\": container with ID starting with 2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a not found: ID does not exist" containerID="2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a" Oct 30 02:19:31.673250 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.672057 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a"} err="failed to get container status \"2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a\": rpc error: code = NotFound desc = could not find container \"2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a\": container with ID starting with 2285c254ceca3d4cb254ebb57c276f109e3e6e88a393e5458e5a7c3f7d96941a not found: ID does not exist" Oct 30 02:19:31.673250 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.672076 5055 scope.go:122] "RemoveContainer" containerID="be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e" Oct 30 02:19:31.683372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.683349 5055 scope.go:122] "RemoveContainer" containerID="be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e" Oct 30 02:19:31.683700 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:19:31.683673 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e\": container with ID starting with be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e not found: ID does not exist" containerID="be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e" Oct 30 02:19:31.683802 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.683706 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e"} err="failed to get container status \"be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e\": rpc error: code = NotFound desc = could not find container \"be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e\": container with ID starting with be22be23982771be376c5473d7739919a33bd0573d5edcdf1442a9cef7a97c6e not found: ID does not exist" Oct 30 02:19:31.742343 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.741426 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7"] Oct 30 02:19:31.745152 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.745125 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c8d376-69bf9d4dfd-t6ml7"] Oct 30 02:19:31.803630 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.803590 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-maas-api-tls\") pod \"3337f877-9bf8-4787-b840-3a07f57c330f\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " Oct 30 02:19:31.804135 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.803649 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3337f877-9bf8-4787-b840-3a07f57c330f-kube-api-access-wrnzh\" (UniqueName: \"kubernetes.io/projected/3337f877-9bf8-4787-b840-3a07f57c330f-kube-api-access-wrnzh\") pod \"3337f877-9bf8-4787-b840-3a07f57c330f\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " Oct 30 02:19:31.804135 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.803710 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3337f877-9bf8-4787-b840-3a07f57c330f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3337f877-9bf8-4787-b840-3a07f57c330f-openshift-service-ca\") pod \"3337f877-9bf8-4787-b840-3a07f57c330f\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " Oct 30 02:19:31.804135 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.803778 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-metrics-certs\") pod \"3337f877-9bf8-4787-b840-3a07f57c330f\" (UID: \"3337f877-9bf8-4787-b840-3a07f57c330f\") " Oct 30 02:19:31.804135 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.804097 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3337f877-9bf8-4787-b840-3a07f57c330f-openshift-service-ca" pod "3337f877-9bf8-4787-b840-3a07f57c330f" (UID: "3337f877-9bf8-4787-b840-3a07f57c330f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:31.804374 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.804291 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3337f877-9bf8-4787-b840-3a07f57c330f-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:31.805974 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.805942 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-maas-api-tls" pod "3337f877-9bf8-4787-b840-3a07f57c330f" (UID: "3337f877-9bf8-4787-b840-3a07f57c330f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:31.806426 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.806401 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-metrics-certs" pod "3337f877-9bf8-4787-b840-3a07f57c330f" (UID: "3337f877-9bf8-4787-b840-3a07f57c330f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:31.806557 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.806441 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3337f877-9bf8-4787-b840-3a07f57c330f-kube-api-access-wrnzh" pod "3337f877-9bf8-4787-b840-3a07f57c330f" (UID: "3337f877-9bf8-4787-b840-3a07f57c330f"). InnerVolumeSpecName "kube-api-access-wrnzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:31.905288 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.905257 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:31.905288 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.905285 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3337f877-9bf8-4787-b840-3a07f57c330f-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:31.905288 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:31.905294 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrnzh\" (UniqueName: \"kubernetes.io/projected/3337f877-9bf8-4787-b840-3a07f57c330f-kube-api-access-wrnzh\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:32.008088 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:32.008049 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx"] Oct 30 02:19:32.010680 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:32.010656 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-340ae6-79fc9db8bc-5t9gx"] Oct 30 02:19:32.819902 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:32.819866 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3337f877-9bf8-4787-b840-3a07f57c330f" path="/var/lib/kubelet/pods/3337f877-9bf8-4787-b840-3a07f57c330f/volumes" Oct 30 02:19:32.820301 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:32.820252 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f07429-1516-4228-85f1-1c37e3100ad7" path="/var/lib/kubelet/pods/b6f07429-1516-4228-85f1-1c37e3100ad7/volumes" Oct 30 02:19:35.112718 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.112676 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" containerID="cri-o://a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354" gracePeriod=45 Oct 30 02:19:35.373726 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.373643 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 02:19:35.546813 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.546780 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tmp-dir\") pod \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " Oct 30 02:19:35.546976 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.546871 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-model-cache\") pod \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " Oct 30 02:19:35.546976 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.546922 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-kube-api-access-j98vz\" (UniqueName: \"kubernetes.io/projected/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-kube-api-access-j98vz\") pod \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " Oct 30 02:19:35.546976 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.546966 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-dshm\") pod \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " Oct 30 02:19:35.547110 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.547014 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tls-certs\") pod \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " Oct 30 02:19:35.547110 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.547070 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tmp-dir" pod "9ba9b42b-7151-450b-9287-e2a2f8bc4deb" (UID: "9ba9b42b-7151-450b-9287-e2a2f8bc4deb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.547110 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.547086 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-home\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-home\") pod \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\" (UID: \"9ba9b42b-7151-450b-9287-e2a2f8bc4deb\") " Oct 30 02:19:35.547268 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.547167 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-model-cache" pod "9ba9b42b-7151-450b-9287-e2a2f8bc4deb" (UID: "9ba9b42b-7151-450b-9287-e2a2f8bc4deb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.547411 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.547383 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-home" pod "9ba9b42b-7151-450b-9287-e2a2f8bc4deb" (UID: "9ba9b42b-7151-450b-9287-e2a2f8bc4deb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.547550 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.547507 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.547550 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.547519 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.547550 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.547528 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.549338 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.549304 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-kube-api-access-j98vz" pod "9ba9b42b-7151-450b-9287-e2a2f8bc4deb" (UID: "9ba9b42b-7151-450b-9287-e2a2f8bc4deb"). InnerVolumeSpecName "kube-api-access-j98vz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:35.549431 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.549367 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tls-certs" pod "9ba9b42b-7151-450b-9287-e2a2f8bc4deb" (UID: "9ba9b42b-7151-450b-9287-e2a2f8bc4deb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:35.549431 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.549379 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-dshm" pod "9ba9b42b-7151-450b-9287-e2a2f8bc4deb" (UID: "9ba9b42b-7151-450b-9287-e2a2f8bc4deb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.648142 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.648052 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j98vz\" (UniqueName: \"kubernetes.io/projected/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-kube-api-access-j98vz\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.648142 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.648086 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.648142 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.648097 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9ba9b42b-7151-450b-9287-e2a2f8bc4deb-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.688501 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.688466 5055 generic.go:356] "Generic (PLEG): container finished" podID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerID="a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354" exitCode=0 Oct 30 02:19:35.688695 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.688508 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" event={"ID":"9ba9b42b-7151-450b-9287-e2a2f8bc4deb","Type":"ContainerDied","Data":"a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354"} Oct 30 02:19:35.688695 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.688511 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" Oct 30 02:19:35.688695 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.688543 5055 scope.go:122] "RemoveContainer" containerID="a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354" Oct 30 02:19:35.688695 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.688533 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw" event={"ID":"9ba9b42b-7151-450b-9287-e2a2f8bc4deb","Type":"ContainerDied","Data":"588e3ce836f998c9435d4eeaa307bb81d33336c2af6b206d43106e297da078b1"} Oct 30 02:19:35.699804 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.699781 5055 scope.go:122] "RemoveContainer" containerID="a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354" Oct 30 02:19:35.700085 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:19:35.700061 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354\": container with ID starting with a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354 not found: ID does not exist" containerID="a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354" Oct 30 02:19:35.700177 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.700092 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354"} err="failed to get container status \"a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354\": rpc error: code = NotFound desc = could not find container \"a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354\": container with ID starting with a972c91a565379494b2193e84cc435e9e4633d01af334b01ab76997ca0666354 not found: ID does not exist" Oct 30 02:19:35.740167 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.740132 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw"] Oct 30 02:19:35.752403 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:35.752366 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-558327/simulator-w3-558327-kserve-695db7695d-nl9qw"] Oct 30 02:19:36.820371 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:36.820307 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" path="/var/lib/kubelet/pods/9ba9b42b-7151-450b-9287-e2a2f8bc4deb/volumes" Oct 30 02:19:43.763339 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:43.763288 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s"] Oct 30 02:19:43.793039 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:43.793002 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s"] Oct 30 02:19:49.221128 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:19:49.221083 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:19:58.785227 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:58.785178 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" containerID="cri-o://0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c" gracePeriod=45 Oct 30 02:19:58.818310 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:58.818269 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" containerID="cri-o://46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43" gracePeriod=45 Oct 30 02:19:58.899499 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:58.899455 5055 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": dial tcp 10.132.0.87:8000: connect: connection refused" Oct 30 02:19:59.128646 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.128593 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 02:19:59.185529 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.185450 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 02:19:59.262401 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.262354 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfb150d4-d9ce-4574-b245-0248f5f4062f-tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb150d4-d9ce-4574-b245-0248f5f4062f-tls-certs\") pod \"cfb150d4-d9ce-4574-b245-0248f5f4062f\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " Oct 30 02:19:59.262599 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.262484 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-tmp-dir\") pod \"cfb150d4-d9ce-4574-b245-0248f5f4062f\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " Oct 30 02:19:59.262668 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.262599 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb150d4-d9ce-4574-b245-0248f5f4062f-kube-api-access-6lpxw\" (UniqueName: \"kubernetes.io/projected/cfb150d4-d9ce-4574-b245-0248f5f4062f-kube-api-access-6lpxw\") pod \"cfb150d4-d9ce-4574-b245-0248f5f4062f\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " Oct 30 02:19:59.262668 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.262632 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-home\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-home\") pod \"cfb150d4-d9ce-4574-b245-0248f5f4062f\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " Oct 30 02:19:59.262792 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.262717 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-dshm\") pod \"cfb150d4-d9ce-4574-b245-0248f5f4062f\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " Oct 30 02:19:59.262792 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.262753 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-model-cache\") pod \"cfb150d4-d9ce-4574-b245-0248f5f4062f\" (UID: \"cfb150d4-d9ce-4574-b245-0248f5f4062f\") " Oct 30 02:19:59.263005 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.262976 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-home" pod "cfb150d4-d9ce-4574-b245-0248f5f4062f" (UID: "cfb150d4-d9ce-4574-b245-0248f5f4062f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.263196 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.263181 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.263253 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.263230 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-model-cache" pod "cfb150d4-d9ce-4574-b245-0248f5f4062f" (UID: "cfb150d4-d9ce-4574-b245-0248f5f4062f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.263492 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.263469 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-tmp-dir" pod "cfb150d4-d9ce-4574-b245-0248f5f4062f" (UID: "cfb150d4-d9ce-4574-b245-0248f5f4062f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.265412 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.265378 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-dshm" pod "cfb150d4-d9ce-4574-b245-0248f5f4062f" (UID: "cfb150d4-d9ce-4574-b245-0248f5f4062f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.266923 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.265729 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfb150d4-d9ce-4574-b245-0248f5f4062f-tls-certs" pod "cfb150d4-d9ce-4574-b245-0248f5f4062f" (UID: "cfb150d4-d9ce-4574-b245-0248f5f4062f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:59.266923 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.265997 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb150d4-d9ce-4574-b245-0248f5f4062f-kube-api-access-6lpxw" pod "cfb150d4-d9ce-4574-b245-0248f5f4062f" (UID: "cfb150d4-d9ce-4574-b245-0248f5f4062f"). InnerVolumeSpecName "kube-api-access-6lpxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:59.364122 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.363935 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-dshm\") pod \"4a970cad-24eb-47e8-ab37-623356dab2e4\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " Oct 30 02:19:59.364122 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364001 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a970cad-24eb-47e8-ab37-623356dab2e4-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a970cad-24eb-47e8-ab37-623356dab2e4-tls-certs\") pod \"4a970cad-24eb-47e8-ab37-623356dab2e4\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " Oct 30 02:19:59.364394 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364162 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-home\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-home\") pod \"4a970cad-24eb-47e8-ab37-623356dab2e4\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " Oct 30 02:19:59.364394 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364223 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a970cad-24eb-47e8-ab37-623356dab2e4-kube-api-access-vlnv5\" (UniqueName: \"kubernetes.io/projected/4a970cad-24eb-47e8-ab37-623356dab2e4-kube-api-access-vlnv5\") pod \"4a970cad-24eb-47e8-ab37-623356dab2e4\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " Oct 30 02:19:59.364394 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364272 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-tmp-dir\") pod \"4a970cad-24eb-47e8-ab37-623356dab2e4\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " Oct 30 02:19:59.364394 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364314 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-model-cache\") pod \"4a970cad-24eb-47e8-ab37-623356dab2e4\" (UID: \"4a970cad-24eb-47e8-ab37-623356dab2e4\") " Oct 30 02:19:59.365195 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364616 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-home" pod "4a970cad-24eb-47e8-ab37-623356dab2e4" (UID: "4a970cad-24eb-47e8-ab37-623356dab2e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.365195 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364805 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.365195 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364948 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.365195 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364963 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb150d4-d9ce-4574-b245-0248f5f4062f-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.365195 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364976 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb150d4-d9ce-4574-b245-0248f5f4062f-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.365195 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.364988 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.365195 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.365000 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpxw\" (UniqueName: \"kubernetes.io/projected/cfb150d4-d9ce-4574-b245-0248f5f4062f-kube-api-access-6lpxw\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.365195 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.365131 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-tmp-dir" pod "4a970cad-24eb-47e8-ab37-623356dab2e4" (UID: "4a970cad-24eb-47e8-ab37-623356dab2e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.366127 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.365432 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-model-cache" pod "4a970cad-24eb-47e8-ab37-623356dab2e4" (UID: "4a970cad-24eb-47e8-ab37-623356dab2e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.366865 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.366839 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-dshm" pod "4a970cad-24eb-47e8-ab37-623356dab2e4" (UID: "4a970cad-24eb-47e8-ab37-623356dab2e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.367409 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.367383 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a970cad-24eb-47e8-ab37-623356dab2e4-tls-certs" pod "4a970cad-24eb-47e8-ab37-623356dab2e4" (UID: "4a970cad-24eb-47e8-ab37-623356dab2e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:59.368100 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.368080 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a970cad-24eb-47e8-ab37-623356dab2e4-kube-api-access-vlnv5" pod "4a970cad-24eb-47e8-ab37-623356dab2e4" (UID: "4a970cad-24eb-47e8-ab37-623356dab2e4"). InnerVolumeSpecName "kube-api-access-vlnv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:59.465798 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.465761 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.465798 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.465789 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a970cad-24eb-47e8-ab37-623356dab2e4-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.465798 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.465799 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlnv5\" (UniqueName: \"kubernetes.io/projected/4a970cad-24eb-47e8-ab37-623356dab2e4-kube-api-access-vlnv5\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.465798 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.465809 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.466248 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.465819 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a970cad-24eb-47e8-ab37-623356dab2e4-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.847039 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.847008 5055 generic.go:356] "Generic (PLEG): container finished" podID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerID="0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c" exitCode=0 Oct 30 02:19:59.847515 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.847050 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" Oct 30 02:19:59.847515 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.847099 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" event={"ID":"cfb150d4-d9ce-4574-b245-0248f5f4062f","Type":"ContainerDied","Data":"0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c"} Oct 30 02:19:59.847515 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.847140 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s" event={"ID":"cfb150d4-d9ce-4574-b245-0248f5f4062f","Type":"ContainerDied","Data":"4c9ad9f75d9a4f84d088c59e8475d4b127ec800299f99aa5f0623c627da01b61"} Oct 30 02:19:59.847515 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.847163 5055 scope.go:122] "RemoveContainer" containerID="0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c" Oct 30 02:19:59.848617 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.848597 5055 generic.go:356] "Generic (PLEG): container finished" podID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerID="46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43" exitCode=0 Oct 30 02:19:59.848673 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.848642 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" Oct 30 02:19:59.848673 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.848650 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" event={"ID":"4a970cad-24eb-47e8-ab37-623356dab2e4","Type":"ContainerDied","Data":"46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43"} Oct 30 02:19:59.848771 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.848676 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s" event={"ID":"4a970cad-24eb-47e8-ab37-623356dab2e4","Type":"ContainerDied","Data":"acce4b707391aa778df6ebc7ba3a556065b690a7fc0b78a5eb861a8a4439904b"} Oct 30 02:19:59.859280 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.859260 5055 scope.go:122] "RemoveContainer" containerID="0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c" Oct 30 02:19:59.859652 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:19:59.859615 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c\": container with ID starting with 0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c not found: ID does not exist" containerID="0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c" Oct 30 02:19:59.859763 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.859654 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c"} err="failed to get container status \"0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c\": rpc error: code = NotFound desc = could not find container \"0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c\": container with ID starting with 0b559e856b4ad7f44af76a8c1266bd990654ab76366040d486ac05d8ddb7652c not found: ID does not exist" Oct 30 02:19:59.859763 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.859678 5055 scope.go:122] "RemoveContainer" containerID="46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43" Oct 30 02:19:59.869076 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.869056 5055 scope.go:122] "RemoveContainer" containerID="46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43" Oct 30 02:19:59.869500 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:19:59.869474 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43\": container with ID starting with 46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43 not found: ID does not exist" containerID="46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43" Oct 30 02:19:59.869611 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.869512 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43"} err="failed to get container status \"46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43\": rpc error: code = NotFound desc = could not find container \"46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43\": container with ID starting with 46e1616c273b304b66ce0ed3d68a850e76d7114fde56e04e7744744e05840b43 not found: ID does not exist" Oct 30 02:19:59.899256 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.899222 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s"] Oct 30 02:19:59.912305 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.912275 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/distinct-2-w0-a3a2ac-kserve-79c97d9865-99b5s"] Oct 30 02:19:59.943756 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.943721 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s"] Oct 30 02:19:59.948298 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:19:59.948267 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/simulator-w0-a3a2ac-kserve-776987ff8-s5v8s"] Oct 30 02:20:00.821446 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:00.821399 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" path="/var/lib/kubelet/pods/4a970cad-24eb-47e8-ab37-623356dab2e4/volumes" Oct 30 02:20:00.822045 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:00.822022 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" path="/var/lib/kubelet/pods/cfb150d4-d9ce-4574-b245-0248f5f4062f/volumes" Oct 30 02:20:14.551088 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:14.551045 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt"] Oct 30 02:20:14.551580 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:14.551310 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" podUID="0d144a49-a6b5-4282-b2af-b7945045301a" containerName="istio-proxy" containerID="cri-o://234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c" gracePeriod=30 Oct 30 02:20:19.222215 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:20:19.222171 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:20:19.847749 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.847687 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 02:20:19.898318 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.898272 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-socket\") pod \"0d144a49-a6b5-4282-b2af-b7945045301a\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " Oct 30 02:20:19.898512 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.898340 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-data\") pod \"0d144a49-a6b5-4282-b2af-b7945045301a\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " Oct 30 02:20:19.898512 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.898368 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-envoy\") pod \"0d144a49-a6b5-4282-b2af-b7945045301a\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " Oct 30 02:20:19.898512 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.898414 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-certs\") pod \"0d144a49-a6b5-4282-b2af-b7945045301a\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " Oct 30 02:20:19.898512 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.898472 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-credential-socket\") pod \"0d144a49-a6b5-4282-b2af-b7945045301a\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " Oct 30 02:20:19.898866 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.898523 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-istio-token\" (UniqueName: \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-istio-token\") pod \"0d144a49-a6b5-4282-b2af-b7945045301a\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " Oct 30 02:20:19.898866 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.898566 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d144a49-a6b5-4282-b2af-b7945045301a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d144a49-a6b5-4282-b2af-b7945045301a-istiod-ca-cert\") pod \"0d144a49-a6b5-4282-b2af-b7945045301a\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " Oct 30 02:20:19.898866 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.898611 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d144a49-a6b5-4282-b2af-b7945045301a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d144a49-a6b5-4282-b2af-b7945045301a-istio-podinfo\") pod \"0d144a49-a6b5-4282-b2af-b7945045301a\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " Oct 30 02:20:19.898866 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.898647 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-kube-api-access-zfhtv\" (UniqueName: \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-kube-api-access-zfhtv\") pod \"0d144a49-a6b5-4282-b2af-b7945045301a\" (UID: \"0d144a49-a6b5-4282-b2af-b7945045301a\") " Oct 30 02:20:19.898866 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.898812 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-certs" pod "0d144a49-a6b5-4282-b2af-b7945045301a" (UID: "0d144a49-a6b5-4282-b2af-b7945045301a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:19.899246 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.899218 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:19.899392 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.899242 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-data" pod "0d144a49-a6b5-4282-b2af-b7945045301a" (UID: "0d144a49-a6b5-4282-b2af-b7945045301a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:19.899392 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.899370 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-credential-socket" pod "0d144a49-a6b5-4282-b2af-b7945045301a" (UID: "0d144a49-a6b5-4282-b2af-b7945045301a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:19.899579 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.899550 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d144a49-a6b5-4282-b2af-b7945045301a-istiod-ca-cert" pod "0d144a49-a6b5-4282-b2af-b7945045301a" (UID: "0d144a49-a6b5-4282-b2af-b7945045301a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:19.899794 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.899770 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-socket" pod "0d144a49-a6b5-4282-b2af-b7945045301a" (UID: "0d144a49-a6b5-4282-b2af-b7945045301a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:19.902073 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.902039 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d144a49-a6b5-4282-b2af-b7945045301a-istio-podinfo" pod "0d144a49-a6b5-4282-b2af-b7945045301a" (UID: "0d144a49-a6b5-4282-b2af-b7945045301a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:20:19.905702 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.905681 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-istio-token" pod "0d144a49-a6b5-4282-b2af-b7945045301a" (UID: "0d144a49-a6b5-4282-b2af-b7945045301a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:19.905817 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.905755 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-kube-api-access-zfhtv" pod "0d144a49-a6b5-4282-b2af-b7945045301a" (UID: "0d144a49-a6b5-4282-b2af-b7945045301a"). InnerVolumeSpecName "kube-api-access-zfhtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:19.906931 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.906911 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-envoy" pod "0d144a49-a6b5-4282-b2af-b7945045301a" (UID: "0d144a49-a6b5-4282-b2af-b7945045301a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:19.969983 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.969949 5055 generic.go:356] "Generic (PLEG): container finished" podID="0d144a49-a6b5-4282-b2af-b7945045301a" containerID="234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c" exitCode=0 Oct 30 02:20:19.970156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.970004 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" Oct 30 02:20:19.970156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.970030 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" event={"ID":"0d144a49-a6b5-4282-b2af-b7945045301a","Type":"ContainerDied","Data":"234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c"} Oct 30 02:20:19.970156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.970085 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt" event={"ID":"0d144a49-a6b5-4282-b2af-b7945045301a","Type":"ContainerDied","Data":"a64c17c6395433d766e26d8d6a45d847105ed8c4108bce30456c5388243a28f8"} Oct 30 02:20:19.970156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.970106 5055 scope.go:122] "RemoveContainer" containerID="234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c" Oct 30 02:20:19.981228 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.981204 5055 scope.go:122] "RemoveContainer" containerID="28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f" Oct 30 02:20:19.990690 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.990670 5055 scope.go:122] "RemoveContainer" containerID="234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c" Oct 30 02:20:19.990995 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:20:19.990976 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c\": container with ID starting with 234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c not found: ID does not exist" containerID="234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c" Oct 30 02:20:19.991049 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.991006 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c"} err="failed to get container status \"234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c\": rpc error: code = NotFound desc = could not find container \"234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c\": container with ID starting with 234e530803125929464f17110d465c3741846978336ba4371499987c70779b7c not found: ID does not exist" Oct 30 02:20:19.991049 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.991026 5055 scope.go:122] "RemoveContainer" containerID="28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f" Oct 30 02:20:19.991275 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:20:19.991252 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f\": container with ID starting with 28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f not found: ID does not exist" containerID="28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f" Oct 30 02:20:19.991400 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:19.991279 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f"} err="failed to get container status \"28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f\": rpc error: code = NotFound desc = could not find container \"28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f\": container with ID starting with 28e0cbd13e20e6d600c71d3cc989656ce4d1abc43cdee8d193af6823b047578f not found: ID does not exist" Oct 30 02:20:20.000033 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.000004 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfhtv\" (UniqueName: \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-kube-api-access-zfhtv\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.000156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.000038 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.000156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.000054 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.000156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.000063 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.000156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.000071 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d144a49-a6b5-4282-b2af-b7945045301a-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.000156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.000079 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d144a49-a6b5-4282-b2af-b7945045301a-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.000156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.000088 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d144a49-a6b5-4282-b2af-b7945045301a-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.000156 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.000096 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d144a49-a6b5-4282-b2af-b7945045301a-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:20.022822 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.022791 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt"] Oct 30 02:20:20.029231 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.029206 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-gr8gt"] Oct 30 02:20:20.824481 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:20.824444 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d144a49-a6b5-4282-b2af-b7945045301a" path="/var/lib/kubelet/pods/0d144a49-a6b5-4282-b2af-b7945045301a/volumes" Oct 30 02:20:22.058563 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.058522 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4"] Oct 30 02:20:22.059100 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.058779 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" podUID="b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e" containerName="maas-api" containerID="cri-o://562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a" gracePeriod=30 Oct 30 02:20:22.366732 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.366700 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 02:20:22.429388 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.425812 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-maas-api-tls\") pod \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " Oct 30 02:20:22.429388 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.425955 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-openshift-service-ca\") pod \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " Oct 30 02:20:22.429388 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.426036 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-metrics-certs\") pod \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " Oct 30 02:20:22.429388 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.426091 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-kube-api-access-28j2l\" (UniqueName: \"kubernetes.io/projected/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-kube-api-access-28j2l\") pod \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\" (UID: \"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e\") " Oct 30 02:20:22.432611 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.432573 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-openshift-service-ca" pod "b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e" (UID: "b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:22.434252 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.434206 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:22.440491 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.440460 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-metrics-certs" pod "b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e" (UID: "b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:22.440491 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.440480 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-maas-api-tls" pod "b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e" (UID: "b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:22.440873 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.440851 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-kube-api-access-28j2l" pod "b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e" (UID: "b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e"). InnerVolumeSpecName "kube-api-access-28j2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:22.535418 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.535384 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:22.535418 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.535413 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28j2l\" (UniqueName: \"kubernetes.io/projected/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-kube-api-access-28j2l\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:22.535418 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.535424 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:22.989114 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.989074 5055 generic.go:356] "Generic (PLEG): container finished" podID="b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e" containerID="562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a" exitCode=0 Oct 30 02:20:22.989306 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.989150 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" Oct 30 02:20:22.989306 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.989162 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" event={"ID":"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e","Type":"ContainerDied","Data":"562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a"} Oct 30 02:20:22.989306 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.989243 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4" event={"ID":"b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e","Type":"ContainerDied","Data":"e53bc2577b7f056a62497c41a482f27afba06327187eb942366ba93e9ced2c43"} Oct 30 02:20:22.989306 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.989266 5055 scope.go:122] "RemoveContainer" containerID="562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a" Oct 30 02:20:22.999222 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.999180 5055 scope.go:122] "RemoveContainer" containerID="562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a" Oct 30 02:20:22.999586 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:20:22.999553 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a\": container with ID starting with 562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a not found: ID does not exist" containerID="562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a" Oct 30 02:20:22.999691 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:22.999598 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a"} err="failed to get container status \"562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a\": rpc error: code = NotFound desc = could not find container \"562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a\": container with ID starting with 562752aaac9b260514e23baef7697cbf27b0cad385d9592b7e8161a0cb0c207a not found: ID does not exist" Oct 30 02:20:23.042836 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:23.042799 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4"] Oct 30 02:20:23.055059 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:23.055017 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-92f0cc-7949956487-mjth4"] Oct 30 02:20:24.818850 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:24.818819 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e" path="/var/lib/kubelet/pods/b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e/volumes" Oct 30 02:20:29.918381 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:29.918319 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm"] Oct 30 02:20:29.918977 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:29.918646 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerName="istio-proxy" containerID="cri-o://d614131fa1de24eaa1c0c708b4fa41b8fa4b36579f1d7d82d4ac71154f6e2bd8" gracePeriod=30 Oct 30 02:20:35.067999 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.067964 5055 generic.go:356] "Generic (PLEG): container finished" podID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerID="d614131fa1de24eaa1c0c708b4fa41b8fa4b36579f1d7d82d4ac71154f6e2bd8" exitCode=0 Oct 30 02:20:35.068565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.068038 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" event={"ID":"da527b6b-9fe4-4c75-8fa4-e99b76442b1a","Type":"ContainerDied","Data":"d614131fa1de24eaa1c0c708b4fa41b8fa4b36579f1d7d82d4ac71154f6e2bd8"} Oct 30 02:20:35.068565 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.068171 5055 scope.go:122] "RemoveContainer" containerID="433715bc198611f37054f549f46520847e653e9fc936a6ae0b4b7163154a979b" Oct 30 02:20:35.201283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.201209 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 02:20:35.371473 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.371434 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-envoy\") pod \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " Oct 30 02:20:35.371657 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.371544 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-credential-socket\") pod \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " Oct 30 02:20:35.371657 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.371581 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-token\" (UniqueName: \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-token\") pod \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " Oct 30 02:20:35.371657 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.371629 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-podinfo\") pod \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " Oct 30 02:20:35.371806 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.371659 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-socket\") pod \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " Oct 30 02:20:35.371806 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.371698 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istiod-ca-cert\") pod \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " Oct 30 02:20:35.371806 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.371748 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-data\") pod \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " Oct 30 02:20:35.371980 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.371804 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-certs\") pod \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " Oct 30 02:20:35.371980 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.371834 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-kube-api-access-t4djw\" (UniqueName: \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-kube-api-access-t4djw\") pod \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\" (UID: \"da527b6b-9fe4-4c75-8fa4-e99b76442b1a\") " Oct 30 02:20:35.371980 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.371922 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-credential-socket" pod "da527b6b-9fe4-4c75-8fa4-e99b76442b1a" (UID: "da527b6b-9fe4-4c75-8fa4-e99b76442b1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:35.372140 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.372023 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-socket" pod "da527b6b-9fe4-4c75-8fa4-e99b76442b1a" (UID: "da527b6b-9fe4-4c75-8fa4-e99b76442b1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:35.372195 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.372146 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-data" pod "da527b6b-9fe4-4c75-8fa4-e99b76442b1a" (UID: "da527b6b-9fe4-4c75-8fa4-e99b76442b1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:35.372254 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.372193 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-certs" pod "da527b6b-9fe4-4c75-8fa4-e99b76442b1a" (UID: "da527b6b-9fe4-4c75-8fa4-e99b76442b1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:35.372400 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.372374 5055 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-credential-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:35.372489 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.372399 5055 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-socket\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:35.372489 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.372413 5055 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-data\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:35.372489 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.372453 5055 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-workload-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:35.372781 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.372753 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istiod-ca-cert" pod "da527b6b-9fe4-4c75-8fa4-e99b76442b1a" (UID: "da527b6b-9fe4-4c75-8fa4-e99b76442b1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:35.374083 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.374061 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-podinfo" pod "da527b6b-9fe4-4c75-8fa4-e99b76442b1a" (UID: "da527b6b-9fe4-4c75-8fa4-e99b76442b1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:20:35.374349 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.374309 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-kube-api-access-t4djw" pod "da527b6b-9fe4-4c75-8fa4-e99b76442b1a" (UID: "da527b6b-9fe4-4c75-8fa4-e99b76442b1a"). InnerVolumeSpecName "kube-api-access-t4djw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:35.374438 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.374359 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-envoy" pod "da527b6b-9fe4-4c75-8fa4-e99b76442b1a" (UID: "da527b6b-9fe4-4c75-8fa4-e99b76442b1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:35.374477 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.374427 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-token" pod "da527b6b-9fe4-4c75-8fa4-e99b76442b1a" (UID: "da527b6b-9fe4-4c75-8fa4-e99b76442b1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:35.473708 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.473669 5055 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-token\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:35.473708 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.473700 5055 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-podinfo\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:35.473708 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.473709 5055 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istiod-ca-cert\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:35.473976 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.473718 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4djw\" (UniqueName: \"kubernetes.io/projected/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-kube-api-access-t4djw\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:35.473976 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:35.473730 5055 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da527b6b-9fe4-4c75-8fa4-e99b76442b1a-istio-envoy\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:36.075567 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:36.075529 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" Oct 30 02:20:36.076024 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:36.075587 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm" event={"ID":"da527b6b-9fe4-4c75-8fa4-e99b76442b1a","Type":"ContainerDied","Data":"9f788d88cd72425f4dccc1ad74e230bcb22a0fd4156bcb248593a37e1fab8adf"} Oct 30 02:20:36.076024 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:36.075640 5055 scope.go:122] "RemoveContainer" containerID="d614131fa1de24eaa1c0c708b4fa41b8fa4b36579f1d7d82d4ac71154f6e2bd8" Oct 30 02:20:36.142801 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:36.142762 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm"] Oct 30 02:20:36.155026 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:36.154978 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-92f0cc-openshift-default-794d566bd5-vhsmm"] Oct 30 02:20:36.818820 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:36.818788 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" path="/var/lib/kubelet/pods/da527b6b-9fe4-4c75-8fa4-e99b76442b1a/volumes" Oct 30 02:20:46.001084 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.001050 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq"] Oct 30 02:20:46.001913 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.001880 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" podUID="3c52f82e-cc60-4e4a-84a5-0f224b1c63f0" containerName="maas-api" containerID="cri-o://b6ae10684a05f1443368c2693fe43d1400b805e4717934b33cdf3614c3de0b4d" gracePeriod=30 Oct 30 02:20:46.135708 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.135670 5055 generic.go:356] "Generic (PLEG): container finished" podID="3c52f82e-cc60-4e4a-84a5-0f224b1c63f0" containerID="b6ae10684a05f1443368c2693fe43d1400b805e4717934b33cdf3614c3de0b4d" exitCode=0 Oct 30 02:20:46.135926 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.135757 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" event={"ID":"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0","Type":"ContainerDied","Data":"b6ae10684a05f1443368c2693fe43d1400b805e4717934b33cdf3614c3de0b4d"} Oct 30 02:20:46.296783 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.296751 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 02:20:46.394962 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.394923 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-openshift-service-ca\") pod \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " Oct 30 02:20:46.395159 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.395122 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-maas-api-tls\") pod \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " Oct 30 02:20:46.395229 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.395186 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-kube-api-access-tmphj\" (UniqueName: \"kubernetes.io/projected/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-kube-api-access-tmphj\") pod \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " Oct 30 02:20:46.395229 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.395220 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-metrics-certs\") pod \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\" (UID: \"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0\") " Oct 30 02:20:46.395426 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.395399 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-openshift-service-ca" pod "3c52f82e-cc60-4e4a-84a5-0f224b1c63f0" (UID: "3c52f82e-cc60-4e4a-84a5-0f224b1c63f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:46.395568 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.395548 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:46.397852 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.397820 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-metrics-certs" pod "3c52f82e-cc60-4e4a-84a5-0f224b1c63f0" (UID: "3c52f82e-cc60-4e4a-84a5-0f224b1c63f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:46.398087 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.398065 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-maas-api-tls" pod "3c52f82e-cc60-4e4a-84a5-0f224b1c63f0" (UID: "3c52f82e-cc60-4e4a-84a5-0f224b1c63f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:46.398401 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.398385 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-kube-api-access-tmphj" pod "3c52f82e-cc60-4e4a-84a5-0f224b1c63f0" (UID: "3c52f82e-cc60-4e4a-84a5-0f224b1c63f0"). InnerVolumeSpecName "kube-api-access-tmphj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:46.496730 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.496699 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:46.496730 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.496728 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmphj\" (UniqueName: \"kubernetes.io/projected/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-kube-api-access-tmphj\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:46.496730 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:46.496738 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:20:47.143649 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:47.143563 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" Oct 30 02:20:47.144140 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:47.143559 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq" event={"ID":"3c52f82e-cc60-4e4a-84a5-0f224b1c63f0","Type":"ContainerDied","Data":"f03b3661205e1c58b23a31e749e937fa333bf6895ad499c2745a5f36958cae40"} Oct 30 02:20:47.144140 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:47.143714 5055 scope.go:122] "RemoveContainer" containerID="b6ae10684a05f1443368c2693fe43d1400b805e4717934b33cdf3614c3de0b4d" Oct 30 02:20:47.196449 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:47.196409 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq"] Oct 30 02:20:47.209751 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:47.209715 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-f1edaf-64b887748d-s85tq"] Oct 30 02:20:48.819461 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:20:48.819417 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c52f82e-cc60-4e4a-84a5-0f224b1c63f0" path="/var/lib/kubelet/pods/3c52f82e-cc60-4e4a-84a5-0f224b1c63f0/volumes" Oct 30 02:20:49.220047 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:20:49.220008 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:21:09.724201 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:09.724166 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p"] Oct 30 02:21:09.724783 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:09.724416 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" podUID="bb94ef17-f808-4ec3-8c42-d48160efb641" containerName="maas-api" containerID="cri-o://0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1" gracePeriod=30 Oct 30 02:21:10.017843 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.017738 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 02:21:10.040578 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.040547 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-metrics-certs\") pod \"bb94ef17-f808-4ec3-8c42-d48160efb641\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " Oct 30 02:21:10.040789 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.040619 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-maas-api-tls\") pod \"bb94ef17-f808-4ec3-8c42-d48160efb641\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " Oct 30 02:21:10.040789 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.040721 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb94ef17-f808-4ec3-8c42-d48160efb641-kube-api-access-cmpgq\" (UniqueName: \"kubernetes.io/projected/bb94ef17-f808-4ec3-8c42-d48160efb641-kube-api-access-cmpgq\") pod \"bb94ef17-f808-4ec3-8c42-d48160efb641\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " Oct 30 02:21:10.040789 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.040767 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb94ef17-f808-4ec3-8c42-d48160efb641-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb94ef17-f808-4ec3-8c42-d48160efb641-openshift-service-ca\") pod \"bb94ef17-f808-4ec3-8c42-d48160efb641\" (UID: \"bb94ef17-f808-4ec3-8c42-d48160efb641\") " Oct 30 02:21:10.041257 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.041222 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb94ef17-f808-4ec3-8c42-d48160efb641-openshift-service-ca" pod "bb94ef17-f808-4ec3-8c42-d48160efb641" (UID: "bb94ef17-f808-4ec3-8c42-d48160efb641"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:10.043077 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.043037 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-metrics-certs" pod "bb94ef17-f808-4ec3-8c42-d48160efb641" (UID: "bb94ef17-f808-4ec3-8c42-d48160efb641"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:10.043301 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.043261 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-maas-api-tls" pod "bb94ef17-f808-4ec3-8c42-d48160efb641" (UID: "bb94ef17-f808-4ec3-8c42-d48160efb641"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:10.043572 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.043534 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb94ef17-f808-4ec3-8c42-d48160efb641-kube-api-access-cmpgq" pod "bb94ef17-f808-4ec3-8c42-d48160efb641" (UID: "bb94ef17-f808-4ec3-8c42-d48160efb641"). InnerVolumeSpecName "kube-api-access-cmpgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:10.142339 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.142297 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:10.142339 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.142346 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmpgq\" (UniqueName: \"kubernetes.io/projected/bb94ef17-f808-4ec3-8c42-d48160efb641-kube-api-access-cmpgq\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:10.142554 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.142358 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb94ef17-f808-4ec3-8c42-d48160efb641-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:10.142554 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.142368 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb94ef17-f808-4ec3-8c42-d48160efb641-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:10.275936 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.275850 5055 generic.go:356] "Generic (PLEG): container finished" podID="bb94ef17-f808-4ec3-8c42-d48160efb641" containerID="0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1" exitCode=0 Oct 30 02:21:10.276092 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.275984 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" event={"ID":"bb94ef17-f808-4ec3-8c42-d48160efb641","Type":"ContainerDied","Data":"0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1"} Oct 30 02:21:10.276092 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.276035 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" event={"ID":"bb94ef17-f808-4ec3-8c42-d48160efb641","Type":"ContainerDied","Data":"359aca662f6e829b7803333dd167e69c499c3add7b09e4501bf4283f42eda22a"} Oct 30 02:21:10.276092 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.276064 5055 scope.go:122] "RemoveContainer" containerID="0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1" Oct 30 02:21:10.276213 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.276202 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p" Oct 30 02:21:10.287728 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.287703 5055 scope.go:122] "RemoveContainer" containerID="0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1" Oct 30 02:21:10.288024 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:21:10.287999 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1\": container with ID starting with 0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1 not found: ID does not exist" containerID="0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1" Oct 30 02:21:10.288161 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.288032 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1"} err="failed to get container status \"0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1\": rpc error: code = NotFound desc = could not find container \"0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1\": container with ID starting with 0d78c079fb1ab0981f34b92e0be1d493bb20b816076af1ca247238dc33de33f1 not found: ID does not exist" Oct 30 02:21:10.328798 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.328760 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p"] Oct 30 02:21:10.335820 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.335791 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-abb30f-bddb7f847-wmd7p"] Oct 30 02:21:10.818881 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:10.818850 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb94ef17-f808-4ec3-8c42-d48160efb641" path="/var/lib/kubelet/pods/bb94ef17-f808-4ec3-8c42-d48160efb641/volumes" Oct 30 02:21:19.219558 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:21:19.219520 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:21:28.711179 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:28.711145 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn"] Oct 30 02:21:28.711626 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:28.711416 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" podUID="d208ed75-9ca5-4dfc-9931-0fc4e5276d0e" containerName="maas-api" containerID="cri-o://9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19" gracePeriod=30 Oct 30 02:21:29.011862 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.011836 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 02:21:29.145934 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.145889 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-metrics-certs\") pod \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " Oct 30 02:21:29.146150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.146037 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-kube-api-access-jdxxt\" (UniqueName: \"kubernetes.io/projected/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-kube-api-access-jdxxt\") pod \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " Oct 30 02:21:29.146150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.146069 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-maas-api-tls\") pod \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " Oct 30 02:21:29.146150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.146109 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-openshift-service-ca\") pod \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\" (UID: \"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e\") " Oct 30 02:21:29.146581 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.146548 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-openshift-service-ca" pod "d208ed75-9ca5-4dfc-9931-0fc4e5276d0e" (UID: "d208ed75-9ca5-4dfc-9931-0fc4e5276d0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:29.148381 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.148352 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-maas-api-tls" pod "d208ed75-9ca5-4dfc-9931-0fc4e5276d0e" (UID: "d208ed75-9ca5-4dfc-9931-0fc4e5276d0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:29.148508 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.148387 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-metrics-certs" pod "d208ed75-9ca5-4dfc-9931-0fc4e5276d0e" (UID: "d208ed75-9ca5-4dfc-9931-0fc4e5276d0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:29.148508 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.148393 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-kube-api-access-jdxxt" pod "d208ed75-9ca5-4dfc-9931-0fc4e5276d0e" (UID: "d208ed75-9ca5-4dfc-9931-0fc4e5276d0e"). InnerVolumeSpecName "kube-api-access-jdxxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:29.247230 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.247129 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdxxt\" (UniqueName: \"kubernetes.io/projected/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-kube-api-access-jdxxt\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:29.247230 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.247162 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:29.247230 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.247172 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:29.247230 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.247181 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:29.382553 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.382519 5055 generic.go:356] "Generic (PLEG): container finished" podID="d208ed75-9ca5-4dfc-9931-0fc4e5276d0e" containerID="9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19" exitCode=0 Oct 30 02:21:29.382743 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.382577 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" Oct 30 02:21:29.382743 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.382620 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" event={"ID":"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e","Type":"ContainerDied","Data":"9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19"} Oct 30 02:21:29.382743 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.382674 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn" event={"ID":"d208ed75-9ca5-4dfc-9931-0fc4e5276d0e","Type":"ContainerDied","Data":"1aa0c302d084678cd73ba03265b777d19ee4b95693342d1ee5938990094ae0bc"} Oct 30 02:21:29.382743 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.382696 5055 scope.go:122] "RemoveContainer" containerID="9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19" Oct 30 02:21:29.392361 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.392312 5055 scope.go:122] "RemoveContainer" containerID="9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19" Oct 30 02:21:29.392654 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:21:29.392635 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19\": container with ID starting with 9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19 not found: ID does not exist" containerID="9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19" Oct 30 02:21:29.392745 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.392661 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19"} err="failed to get container status \"9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19\": rpc error: code = NotFound desc = could not find container \"9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19\": container with ID starting with 9bb125549555e6dfb4131c8ad70ae2b0b88ae88d42d66dc077608c94383ebb19 not found: ID does not exist" Oct 30 02:21:29.436346 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.436291 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn"] Oct 30 02:21:29.448549 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:29.448513 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-861cd2-8575f94db4-rrdrn"] Oct 30 02:21:30.821563 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:30.821531 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d208ed75-9ca5-4dfc-9931-0fc4e5276d0e" path="/var/lib/kubelet/pods/d208ed75-9ca5-4dfc-9931-0fc4e5276d0e/volumes" Oct 30 02:21:42.634037 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:42.633998 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx"] Oct 30 02:21:42.646590 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:42.646555 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7"] Oct 30 02:21:49.218869 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:21:49.218821 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:21:57.656170 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:57.656131 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" containerID="cri-o://1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1" gracePeriod=45 Oct 30 02:21:57.668824 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:57.668787 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" containerID="cri-o://a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91" gracePeriod=45 Oct 30 02:21:57.938201 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:57.938170 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 02:21:57.978221 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:57.977973 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 02:21:58.039396 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039355 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-dshm\") pod \"ee3da2e2-7a92-46c9-8bde-902d968766a4\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " Oct 30 02:21:58.039597 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039426 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee3da2e2-7a92-46c9-8bde-902d968766a4-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3da2e2-7a92-46c9-8bde-902d968766a4-tls-certs\") pod \"ee3da2e2-7a92-46c9-8bde-902d968766a4\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " Oct 30 02:21:58.039597 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039451 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-model-cache\") pod \"520cd461-c87c-43a8-acea-4a669a89158e\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " Oct 30 02:21:58.039597 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039499 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-tmp-dir\") pod \"520cd461-c87c-43a8-acea-4a669a89158e\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " Oct 30 02:21:58.039597 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039536 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-tmp-dir\") pod \"ee3da2e2-7a92-46c9-8bde-902d968766a4\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " Oct 30 02:21:58.039597 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039582 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520cd461-c87c-43a8-acea-4a669a89158e-kube-api-access-f5v9w\" (UniqueName: \"kubernetes.io/projected/520cd461-c87c-43a8-acea-4a669a89158e-kube-api-access-f5v9w\") pod \"520cd461-c87c-43a8-acea-4a669a89158e\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " Oct 30 02:21:58.039903 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039617 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee3da2e2-7a92-46c9-8bde-902d968766a4-kube-api-access-g9j8n\" (UniqueName: \"kubernetes.io/projected/ee3da2e2-7a92-46c9-8bde-902d968766a4-kube-api-access-g9j8n\") pod \"ee3da2e2-7a92-46c9-8bde-902d968766a4\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " Oct 30 02:21:58.039903 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039643 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/520cd461-c87c-43a8-acea-4a669a89158e-tls-certs\" (UniqueName: \"kubernetes.io/secret/520cd461-c87c-43a8-acea-4a669a89158e-tls-certs\") pod \"520cd461-c87c-43a8-acea-4a669a89158e\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " Oct 30 02:21:58.039903 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039695 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-home\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-home\") pod \"ee3da2e2-7a92-46c9-8bde-902d968766a4\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " Oct 30 02:21:58.039903 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039731 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-home\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-home\") pod \"520cd461-c87c-43a8-acea-4a669a89158e\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " Oct 30 02:21:58.039903 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039779 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-dshm\") pod \"520cd461-c87c-43a8-acea-4a669a89158e\" (UID: \"520cd461-c87c-43a8-acea-4a669a89158e\") " Oct 30 02:21:58.039903 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.039831 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-model-cache\") pod \"ee3da2e2-7a92-46c9-8bde-902d968766a4\" (UID: \"ee3da2e2-7a92-46c9-8bde-902d968766a4\") " Oct 30 02:21:58.040535 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.040506 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-model-cache" pod "ee3da2e2-7a92-46c9-8bde-902d968766a4" (UID: "ee3da2e2-7a92-46c9-8bde-902d968766a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:58.041051 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.041026 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-home" pod "ee3da2e2-7a92-46c9-8bde-902d968766a4" (UID: "ee3da2e2-7a92-46c9-8bde-902d968766a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:58.042224 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.042128 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-home" pod "520cd461-c87c-43a8-acea-4a669a89158e" (UID: "520cd461-c87c-43a8-acea-4a669a89158e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:58.042224 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.042163 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-tmp-dir" pod "ee3da2e2-7a92-46c9-8bde-902d968766a4" (UID: "ee3da2e2-7a92-46c9-8bde-902d968766a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:58.042513 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.042244 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-tmp-dir" pod "520cd461-c87c-43a8-acea-4a669a89158e" (UID: "520cd461-c87c-43a8-acea-4a669a89158e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:58.042513 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.042492 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-model-cache" pod "520cd461-c87c-43a8-acea-4a669a89158e" (UID: "520cd461-c87c-43a8-acea-4a669a89158e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:58.043545 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.043514 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/520cd461-c87c-43a8-acea-4a669a89158e-tls-certs" pod "520cd461-c87c-43a8-acea-4a669a89158e" (UID: "520cd461-c87c-43a8-acea-4a669a89158e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:58.043994 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.043830 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-dshm" pod "ee3da2e2-7a92-46c9-8bde-902d968766a4" (UID: "ee3da2e2-7a92-46c9-8bde-902d968766a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:58.043994 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.043965 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee3da2e2-7a92-46c9-8bde-902d968766a4-tls-certs" pod "ee3da2e2-7a92-46c9-8bde-902d968766a4" (UID: "ee3da2e2-7a92-46c9-8bde-902d968766a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:58.044407 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.044384 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee3da2e2-7a92-46c9-8bde-902d968766a4-kube-api-access-g9j8n" pod "ee3da2e2-7a92-46c9-8bde-902d968766a4" (UID: "ee3da2e2-7a92-46c9-8bde-902d968766a4"). InnerVolumeSpecName "kube-api-access-g9j8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:58.045058 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.045030 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520cd461-c87c-43a8-acea-4a669a89158e-kube-api-access-f5v9w" pod "520cd461-c87c-43a8-acea-4a669a89158e" (UID: "520cd461-c87c-43a8-acea-4a669a89158e"). InnerVolumeSpecName "kube-api-access-f5v9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:58.045574 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.045551 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-dshm" pod "520cd461-c87c-43a8-acea-4a669a89158e" (UID: "520cd461-c87c-43a8-acea-4a669a89158e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:58.141518 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141473 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141518 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141503 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141518 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141521 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee3da2e2-7a92-46c9-8bde-902d968766a4-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141784 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141532 5055 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-model-cache\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141784 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141540 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141784 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141547 5055 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-tmp-dir\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141784 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141556 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5v9w\" (UniqueName: \"kubernetes.io/projected/520cd461-c87c-43a8-acea-4a669a89158e-kube-api-access-f5v9w\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141784 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141565 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9j8n\" (UniqueName: \"kubernetes.io/projected/ee3da2e2-7a92-46c9-8bde-902d968766a4-kube-api-access-g9j8n\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141784 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141573 5055 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/520cd461-c87c-43a8-acea-4a669a89158e-tls-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141784 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141580 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee3da2e2-7a92-46c9-8bde-902d968766a4-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141784 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141587 5055 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-home\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.141784 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.141594 5055 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/520cd461-c87c-43a8-acea-4a669a89158e-dshm\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.551393 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.551361 5055 generic.go:356] "Generic (PLEG): container finished" podID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerID="a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91" exitCode=0 Oct 30 02:21:58.551600 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.551460 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" Oct 30 02:21:58.551600 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.551498 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" event={"ID":"ee3da2e2-7a92-46c9-8bde-902d968766a4","Type":"ContainerDied","Data":"a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91"} Oct 30 02:21:58.551600 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.551557 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7" event={"ID":"ee3da2e2-7a92-46c9-8bde-902d968766a4","Type":"ContainerDied","Data":"a095a879acfabd247b2aae745432ae22be5ca5f0ae9704004d766800a79bae92"} Oct 30 02:21:58.551600 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.551582 5055 scope.go:122] "RemoveContainer" containerID="a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91" Oct 30 02:21:58.552826 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.552807 5055 generic.go:356] "Generic (PLEG): container finished" podID="520cd461-c87c-43a8-acea-4a669a89158e" containerID="1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1" exitCode=0 Oct 30 02:21:58.552892 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.552853 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" Oct 30 02:21:58.552938 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.552884 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" event={"ID":"520cd461-c87c-43a8-acea-4a669a89158e","Type":"ContainerDied","Data":"1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1"} Oct 30 02:21:58.552938 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.552912 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx" event={"ID":"520cd461-c87c-43a8-acea-4a669a89158e","Type":"ContainerDied","Data":"0038dbb66644325ecf0828ca65ebd187144c46b6e4503923b69250f4f5221620"} Oct 30 02:21:58.562279 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.562258 5055 scope.go:122] "RemoveContainer" containerID="a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91" Oct 30 02:21:58.562814 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:21:58.562699 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91\": container with ID starting with a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91 not found: ID does not exist" containerID="a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91" Oct 30 02:21:58.562814 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.562743 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91"} err="failed to get container status \"a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91\": rpc error: code = NotFound desc = could not find container \"a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91\": container with ID starting with a0f33a30604a77b88209433b6581f5f7daf050498af4711fe14b96fd52abaa91 not found: ID does not exist" Oct 30 02:21:58.562814 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.562766 5055 scope.go:122] "RemoveContainer" containerID="1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1" Oct 30 02:21:58.575468 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.575428 5055 scope.go:122] "RemoveContainer" containerID="1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1" Oct 30 02:21:58.575860 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:21:58.575836 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1\": container with ID starting with 1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1 not found: ID does not exist" containerID="1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1" Oct 30 02:21:58.575969 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.575866 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1"} err="failed to get container status \"1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1\": rpc error: code = NotFound desc = could not find container \"1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1\": container with ID starting with 1382eef883f015290fb3047fec2ba6272dc517ed5a8b01daab2bc659762772a1 not found: ID does not exist" Oct 30 02:21:58.601748 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.601713 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx"] Oct 30 02:21:58.614081 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.614046 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/premium-w1-cf82f8-kserve-6f99b48b66-mn4sx"] Oct 30 02:21:58.638881 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.638850 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7"] Oct 30 02:21:58.642551 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.642524 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/simulator-w1-cf82f8-kserve-565bfb8887-mb8p7"] Oct 30 02:21:58.820045 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.819949 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520cd461-c87c-43a8-acea-4a669a89158e" path="/var/lib/kubelet/pods/520cd461-c87c-43a8-acea-4a669a89158e/volumes" Oct 30 02:21:58.820570 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:21:58.820548 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" path="/var/lib/kubelet/pods/ee3da2e2-7a92-46c9-8bde-902d968766a4/volumes" Oct 30 02:22:06.256652 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.256600 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f"] Oct 30 02:22:06.257150 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.256853 5055 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" podUID="45b20e1b-1132-49c6-a431-e93ccf0e7bdb" containerName="maas-api" containerID="cri-o://e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d" gracePeriod=30 Oct 30 02:22:06.546388 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.546359 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 02:22:06.607086 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.605677 5055 generic.go:356] "Generic (PLEG): container finished" podID="45b20e1b-1132-49c6-a431-e93ccf0e7bdb" containerID="e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d" exitCode=0 Oct 30 02:22:06.607086 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.605811 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" event={"ID":"45b20e1b-1132-49c6-a431-e93ccf0e7bdb","Type":"ContainerDied","Data":"e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d"} Oct 30 02:22:06.607086 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.605858 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" event={"ID":"45b20e1b-1132-49c6-a431-e93ccf0e7bdb","Type":"ContainerDied","Data":"b34db897ab9bab9e5b732c8590681930277d506b7cb986df9c61633394516731"} Oct 30 02:22:06.607086 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.605885 5055 scope.go:122] "RemoveContainer" containerID="e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d" Oct 30 02:22:06.607086 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.606014 5055 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f" Oct 30 02:22:06.627018 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.626976 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-kube-api-access-nrj56\" (UniqueName: \"kubernetes.io/projected/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-kube-api-access-nrj56\") pod \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " Oct 30 02:22:06.627214 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.627058 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-metrics-certs\") pod \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " Oct 30 02:22:06.627214 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.627154 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-openshift-service-ca\") pod \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " Oct 30 02:22:06.627377 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.627250 5055 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-maas-api-tls\") pod \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\" (UID: \"45b20e1b-1132-49c6-a431-e93ccf0e7bdb\") " Oct 30 02:22:06.627697 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.627665 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-openshift-service-ca" pod "45b20e1b-1132-49c6-a431-e93ccf0e7bdb" (UID: "45b20e1b-1132-49c6-a431-e93ccf0e7bdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:06.628236 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.628219 5055 scope.go:122] "RemoveContainer" containerID="e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d" Oct 30 02:22:06.628652 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:22:06.628613 5055 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d\": container with ID starting with e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d not found: ID does not exist" containerID="e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d" Oct 30 02:22:06.628779 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.628650 5055 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d"} err="failed to get container status \"e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d\": rpc error: code = NotFound desc = could not find container \"e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d\": container with ID starting with e78bef5cf2a818f7fed9308f3fb32d464510a070cbc8802ea17944c5d7b81d6d not found: ID does not exist" Oct 30 02:22:06.630138 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.630114 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-metrics-certs" pod "45b20e1b-1132-49c6-a431-e93ccf0e7bdb" (UID: "45b20e1b-1132-49c6-a431-e93ccf0e7bdb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:22:06.630248 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.630181 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-maas-api-tls" pod "45b20e1b-1132-49c6-a431-e93ccf0e7bdb" (UID: "45b20e1b-1132-49c6-a431-e93ccf0e7bdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:22:06.630317 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.630295 5055 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-kube-api-access-nrj56" pod "45b20e1b-1132-49c6-a431-e93ccf0e7bdb" (UID: "45b20e1b-1132-49c6-a431-e93ccf0e7bdb"). InnerVolumeSpecName "kube-api-access-nrj56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:06.728391 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.728321 5055 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrj56\" (UniqueName: \"kubernetes.io/projected/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-kube-api-access-nrj56\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:22:06.728391 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.728387 5055 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-metrics-certs\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:22:06.728391 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.728396 5055 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-openshift-service-ca\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:22:06.728651 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.728407 5055 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45b20e1b-1132-49c6-a431-e93ccf0e7bdb-maas-api-tls\") on node \"ip-10-0-134-254.ec2.internal\" DevicePath \"\"" Oct 30 02:22:06.945391 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.945354 5055 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f"] Oct 30 02:22:06.947085 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:06.947061 5055 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cf82f8-6bbfcb8749-x4v4f"] Oct 30 02:22:08.818776 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:08.818741 5055 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45b20e1b-1132-49c6-a431-e93ccf0e7bdb" path="/var/lib/kubelet/pods/45b20e1b-1132-49c6-a431-e93ccf0e7bdb/volumes" Oct 30 02:22:13.563101 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:13.563071 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:22:13.567857 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:13.567833 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:22:15.396865 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:15.396837 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nz6qt_ee296e37-1ffe-4f80-825b-086fa417666d/manager/0.log" Oct 30 02:22:15.604372 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:15.604350 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hdh5g_7d2c4e13-3895-4202-aef2-b0e726d79e3b/manager/0.log" Oct 30 02:22:15.834257 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:15.834222 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-59kh5_0bf33bf9-2177-45f4-9ebd-493c07fb49db/manager/0.log" Oct 30 02:22:17.429439 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:17.429402 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jhxpw_58571a89-2cf1-44fa-8ef4-a052d02e1a82/manager/0.log" Oct 30 02:22:17.875663 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:17.875634 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cvjxp_36ce2879-a790-49fb-85d7-e6cd921a42c1/manager/0.log" Oct 30 02:22:18.346821 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:18.346781 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78cc6d847c-4nbvj_b505b494-3911-42aa-80f7-71cfece2b6da/manager/0.log" Oct 30 02:22:18.547370 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:18.547311 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cbpcp_5dce57b6-a313-4051-a33b-7300d36d13ab/server/0.log" Oct 30 02:22:18.744814 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:18.744773 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-92htv_a93ddb49-90a6-416d-981a-afd68b09076b/manager/0.log" Oct 30 02:22:18.971523 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:18.971488 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tf79s_916568e2-0b52-47e0-a7bd-3635c0ceca97/proxy/0.log" Oct 30 02:22:19.218375 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:22:19.218341 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:22:30.526705 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:30.526613 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jhxpw_58571a89-2cf1-44fa-8ef4-a052d02e1a82/manager/0.log" Oct 30 02:22:30.965944 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:30.965901 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cvjxp_36ce2879-a790-49fb-85d7-e6cd921a42c1/manager/0.log" Oct 30 02:22:31.434652 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:31.434560 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78cc6d847c-4nbvj_b505b494-3911-42aa-80f7-71cfece2b6da/manager/0.log" Oct 30 02:22:31.631071 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:31.631037 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cbpcp_5dce57b6-a313-4051-a33b-7300d36d13ab/server/0.log" Oct 30 02:22:31.830151 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:31.830119 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-92htv_a93ddb49-90a6-416d-981a-afd68b09076b/manager/0.log" Oct 30 02:22:32.043419 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:32.043388 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tf79s_916568e2-0b52-47e0-a7bd-3635c0ceca97/proxy/0.log" Oct 30 02:22:34.897358 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:34.897313 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-m77kt_a32eb4fc-8b4a-4afa-b95a-29497eee0df4/postgres/0.log" Oct 30 02:22:36.673993 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:36.673960 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-nz6qt_ee296e37-1ffe-4f80-825b-086fa417666d/manager/0.log" Oct 30 02:22:36.879556 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:36.879518 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-hdh5g_7d2c4e13-3895-4202-aef2-b0e726d79e3b/manager/0.log" Oct 30 02:22:37.116806 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:37.116768 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-59kh5_0bf33bf9-2177-45f4-9ebd-493c07fb49db/manager/0.log" Oct 30 02:22:38.348180 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:38.348146 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fvvsbg_551f197c-a32d-4dcb-b4f7-d8106d500e9b/istio-proxy/0.log" Oct 30 02:22:39.570719 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:39.570688 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bvwbz_b367a50a-f9ee-4f39-896b-6baf0615bb8b/istio-proxy/0.log" Oct 30 02:22:40.007404 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:40.007373 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-z65cw_f58c0b93-1a99-44ad-847d-d5022d773a51/istio-proxy/0.log" Oct 30 02:22:40.202654 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:40.202625 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64695b4bc9-6z9hq_4b636b75-528c-4c41-8d2a-082205cbfdbb/istio-proxy/0.log" Oct 30 02:22:40.786091 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:40.786047 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-67bf8c5c5-mtkk5_a310cb37-d20b-4b25-9796-c5f56c44adac/router/0.log" Oct 30 02:22:49.217867 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:22:49.217830 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:22:59.892940 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:22:59.892907 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rz57q_8bc436c0-2b1b-477e-81aa-af1a1eff9779/global-pull-secret-syncer/0.log" Oct 30 02:23:00.055040 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:00.055016 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-4q66s_1dcbad1e-b7e0-4dac-a2bc-55ddf6bb55f5/connection-checker/0.log" Oct 30 02:23:00.175580 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:00.175508 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-c9s9t_4a8853b4-1b14-4b16-81d8-007e7827d8dd/connection-checker/0.log" Oct 30 02:23:00.305227 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:00.305203 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xsnwp_b4e79600-74c8-4e22-8fce-fbd02aca1e38/connection-checker/0.log" Oct 30 02:23:00.863705 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:00.863670 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mvmkl_628af330-6652-4df0-8275-9f691e2c8eb3/konnectivity-agent/0.log" Oct 30 02:23:00.979753 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:00.979726 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-254.ec2.internal_7dddf377448d96e272c684d4e526ba4c/haproxy/0.log" Oct 30 02:23:07.669213 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:07.669180 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-pb877_c9b752e3-e38a-4d0f-babd-7922d6a2e872/cluster-monitoring-operator/0.log" Oct 30 02:23:09.152864 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:09.152831 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vc8rt_907bc83e-74a4-4edb-8cf3-28d0fd1a6963/node-exporter/0.log" Oct 30 02:23:09.275704 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:09.275678 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vc8rt_907bc83e-74a4-4edb-8cf3-28d0fd1a6963/kube-rbac-proxy/0.log" Oct 30 02:23:09.386462 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:09.386425 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vc8rt_907bc83e-74a4-4edb-8cf3-28d0fd1a6963/init-textfile/0.log" Oct 30 02:23:10.818412 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818382 5055 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds"] Oct 30 02:23:10.818828 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818767 5055 cpu_manager.go:414] "containerMap: removing container" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" Oct 30 02:23:10.818828 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818778 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" Oct 30 02:23:10.818828 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818787 5055 cpu_manager.go:414] "containerMap: removing container" podUID="679f8047-1bb0-4123-92d7-6474c85ce2be" containerName="maas-api" Oct 30 02:23:10.818828 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818792 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="679f8047-1bb0-4123-92d7-6474c85ce2be" containerName="maas-api" Oct 30 02:23:10.818828 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818799 5055 cpu_manager.go:414] "containerMap: removing container" podUID="43fbb343-2521-4767-b229-0912ab4880d1" containerName="maas-api" Oct 30 02:23:10.818828 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818803 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fbb343-2521-4767-b229-0912ab4880d1" containerName="maas-api" Oct 30 02:23:10.818828 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818812 5055 cpu_manager.go:414] "containerMap: removing container" podUID="98ae70a5-1283-439e-9b9d-12644878a758" containerName="maas-api" Oct 30 02:23:10.818828 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818816 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ae70a5-1283-439e-9b9d-12644878a758" containerName="maas-api" Oct 30 02:23:10.818828 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818825 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0d144a49-a6b5-4282-b2af-b7945045301a" containerName="istio-proxy" Oct 30 02:23:10.818828 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818829 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d144a49-a6b5-4282-b2af-b7945045301a" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818835 5055 cpu_manager.go:414] "containerMap: removing container" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818840 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818850 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818854 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818860 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b6f07429-1516-4228-85f1-1c37e3100ad7" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818866 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f07429-1516-4228-85f1-1c37e3100ad7" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818876 5055 cpu_manager.go:414] "containerMap: removing container" podUID="0f1019d4-0665-4023-b5fe-52f5f2394723" containerName="payload-pre-processing" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818881 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1019d4-0665-4023-b5fe-52f5f2394723" containerName="payload-pre-processing" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818888 5055 cpu_manager.go:414] "containerMap: removing container" podUID="902d35cf-e127-4756-9a4b-ac9225465e17" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818892 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="902d35cf-e127-4756-9a4b-ac9225465e17" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818900 5055 cpu_manager.go:414] "containerMap: removing container" podUID="49dc7504-c3b9-42da-bfcc-78a51040b9c2" containerName="sim" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818904 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49dc7504-c3b9-42da-bfcc-78a51040b9c2" containerName="sim" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818910 5055 cpu_manager.go:414] "containerMap: removing container" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818915 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818920 5055 cpu_manager.go:414] "containerMap: removing container" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818924 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818930 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818934 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818940 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7c10d615-5cde-47ae-821b-7e37711e71cd" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818944 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c10d615-5cde-47ae-821b-7e37711e71cd" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818953 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a7381e33-dd3e-4243-818d-21413e7e163e" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818956 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7381e33-dd3e-4243-818d-21413e7e163e" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818963 5055 cpu_manager.go:414] "containerMap: removing container" podUID="6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818967 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818974 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818978 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818983 5055 cpu_manager.go:414] "containerMap: removing container" podUID="7baef220-6084-416e-a7c2-c86722049b2f" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818988 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7baef220-6084-416e-a7c2-c86722049b2f" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.818996 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3c52f82e-cc60-4e4a-84a5-0f224b1c63f0" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819000 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c52f82e-cc60-4e4a-84a5-0f224b1c63f0" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819007 5055 cpu_manager.go:414] "containerMap: removing container" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819012 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819018 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819022 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819033 5055 cpu_manager.go:414] "containerMap: removing container" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819037 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819043 5055 cpu_manager.go:414] "containerMap: removing container" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819047 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819055 5055 cpu_manager.go:414] "containerMap: removing container" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819059 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819067 5055 cpu_manager.go:414] "containerMap: removing container" podUID="fee10591-c2b7-4290-a568-e6d39436988f" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819072 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee10591-c2b7-4290-a568-e6d39436988f" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819077 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d208ed75-9ca5-4dfc-9931-0fc4e5276d0e" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819081 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d208ed75-9ca5-4dfc-9931-0fc4e5276d0e" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819087 5055 cpu_manager.go:414] "containerMap: removing container" podUID="bb94ef17-f808-4ec3-8c42-d48160efb641" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819091 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb94ef17-f808-4ec3-8c42-d48160efb641" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819098 5055 cpu_manager.go:414] "containerMap: removing container" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819103 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819108 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819112 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819120 5055 cpu_manager.go:414] "containerMap: removing container" podUID="64ecaa40-4720-4849-8d03-892807164526" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819125 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64ecaa40-4720-4849-8d03-892807164526" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819131 5055 cpu_manager.go:414] "containerMap: removing container" podUID="3337f877-9bf8-4787-b840-3a07f57c330f" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819135 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3337f877-9bf8-4787-b840-3a07f57c330f" containerName="maas-api" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819143 5055 cpu_manager.go:414] "containerMap: removing container" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819147 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819153 5055 cpu_manager.go:414] "containerMap: removing container" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819157 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819164 5055 cpu_manager.go:414] "containerMap: removing container" podUID="baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" containerName="main" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819168 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" containerName="main" Oct 30 02:23:10.819149 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819175 5055 cpu_manager.go:414] "containerMap: removing container" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819180 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819187 5055 cpu_manager.go:414] "containerMap: removing container" podUID="28e91e96-2e4d-4df8-b9fa-928f401194c6" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819191 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e91e96-2e4d-4df8-b9fa-928f401194c6" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819202 5055 cpu_manager.go:414] "containerMap: removing container" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819207 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819214 5055 cpu_manager.go:414] "containerMap: removing container" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819218 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819226 5055 cpu_manager.go:414] "containerMap: removing container" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819230 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819237 5055 cpu_manager.go:414] "containerMap: removing container" podUID="79be7f9d-f615-4c9f-9db8-f20c0f1a6206" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819242 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79be7f9d-f615-4c9f-9db8-f20c0f1a6206" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819248 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819252 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819259 5055 cpu_manager.go:414] "containerMap: removing container" podUID="dec73c66-91e7-4f40-9a01-0080db07ec87" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819262 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dec73c66-91e7-4f40-9a01-0080db07ec87" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819269 5055 cpu_manager.go:414] "containerMap: removing container" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819273 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819279 5055 cpu_manager.go:414] "containerMap: removing container" podUID="e7f5d35f-45a0-4e49-aa98-65836e5d82b5" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819283 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f5d35f-45a0-4e49-aa98-65836e5d82b5" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819289 5055 cpu_manager.go:414] "containerMap: removing container" podUID="45b20e1b-1132-49c6-a431-e93ccf0e7bdb" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819293 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45b20e1b-1132-49c6-a431-e93ccf0e7bdb" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819300 5055 cpu_manager.go:414] "containerMap: removing container" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819304 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819313 5055 cpu_manager.go:414] "containerMap: removing container" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819317 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819340 5055 cpu_manager.go:414] "containerMap: removing container" podUID="65f34b67-7d82-405f-8825-8fede2564631" containerName="payload-processing" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819344 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65f34b67-7d82-405f-8825-8fede2564631" containerName="payload-processing" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819349 5055 cpu_manager.go:414] "containerMap: removing container" podUID="1ae14669-6647-4386-a4fd-7a769705984e" containerName="payload-pre-processing" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819355 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae14669-6647-4386-a4fd-7a769705984e" containerName="payload-pre-processing" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819363 5055 cpu_manager.go:414] "containerMap: removing container" podUID="b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819367 5055 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819428 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="79be7f9d-f615-4c9f-9db8-f20c0f1a6206" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819435 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7888167-f48d-41cb-8a4a-dd9510bb9481" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819443 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d144a49-a6b5-4282-b2af-b7945045301a" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819449 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="dec73c66-91e7-4f40-9a01-0080db07ec87" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819456 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="02cf2cd3-e5cd-4f83-968a-2694debfe092" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819460 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f5d35f-45a0-4e49-aa98-65836e5d82b5" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819466 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="45b20e1b-1132-49c6-a431-e93ccf0e7bdb" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819471 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="488e9de5-05b4-48ed-a23a-f4db797631df" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819477 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819483 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819489 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="65f34b67-7d82-405f-8825-8fede2564631" containerName="payload-processing" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819495 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae14669-6647-4386-a4fd-7a769705984e" containerName="payload-pre-processing" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819503 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c8e1a3-fb92-4dd3-83ba-a60fce315c6e" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819510 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819515 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="679f8047-1bb0-4123-92d7-6474c85ce2be" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819520 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fbb343-2521-4767-b229-0912ab4880d1" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819526 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ae70a5-1283-439e-9b9d-12644878a758" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819530 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6e20bcb-70f4-416a-a159-73e28ad6e931" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819535 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819540 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f07429-1516-4228-85f1-1c37e3100ad7" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819546 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1019d4-0665-4023-b5fe-52f5f2394723" containerName="payload-pre-processing" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819551 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="902d35cf-e127-4756-9a4b-ac9225465e17" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819555 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="49dc7504-c3b9-42da-bfcc-78a51040b9c2" containerName="sim" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819562 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a970cad-24eb-47e8-ab37-623356dab2e4" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819567 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819572 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d58e2b-8457-4d5a-b096-dd5ceb25ca6b" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819578 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c10d615-5cde-47ae-821b-7e37711e71cd" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819583 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7381e33-dd3e-4243-818d-21413e7e163e" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819587 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1c5a9d-8deb-4ebc-bc1e-ad12960134a2" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819593 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58c8f32-415f-40bd-ad3f-275b80c34f9a" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819599 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7baef220-6084-416e-a7c2-c86722049b2f" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819603 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c52f82e-cc60-4e4a-84a5-0f224b1c63f0" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819608 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819613 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0dfd823-2436-423a-b2f6-67a537b6a329" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819620 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb150d4-d9ce-4574-b245-0248f5f4062f" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819626 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="46824cf2-1354-4205-8a62-21f124771343" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819631 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd53b8e7-8d27-49aa-9a7d-21d91a3a6cb6" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819636 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee10591-c2b7-4290-a568-e6d39436988f" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819642 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="d208ed75-9ca5-4dfc-9931-0fc4e5276d0e" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819647 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb94ef17-f808-4ec3-8c42-d48160efb641" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819652 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="79831cb6-7cc2-4153-b501-079ef3bbe670" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819656 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0b9ecdc-93c7-4284-9c50-92ce07e34d90" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819662 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="64ecaa40-4720-4849-8d03-892807164526" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819667 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="3337f877-9bf8-4787-b840-3a07f57c330f" containerName="maas-api" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819671 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="da527b6b-9fe4-4c75-8fa4-e99b76442b1a" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819676 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e7b35c-1ce6-4b22-9c80-f6745b3727b3" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819681 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="baeba5ed-7e13-4ff1-a2b5-2ecb56ca6b4c" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819686 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="a120922c-cb9e-4f13-81aa-0b68ccdf0f9c" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819691 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e91e96-2e4d-4df8-b9fa-928f401194c6" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819696 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba9b42b-7151-450b-9287-e2a2f8bc4deb" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819700 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="520cd461-c87c-43a8-acea-4a669a89158e" containerName="main" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819706 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="7458b5bd-13a6-4044-a5cb-66160c5dd31a" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819710 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc69473a-f21a-4e8a-83da-2f2469a423bd" containerName="istio-proxy" Oct 30 02:23:10.821191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.819715 5055 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee3da2e2-7a92-46c9-8bde-902d968766a4" containerName="main" Oct 30 02:23:10.824137 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.822671 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:10.842140 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.842106 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds"] Oct 30 02:23:10.921299 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.921265 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-6lhwt_6a6e53e8-68df-4699-b745-d2183fb8d473/prometheus-operator-admission-webhook/0.log" Oct 30 02:23:10.929073 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.929040 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9dvcx\"/\"openshift-service-ca.crt\"" Oct 30 02:23:10.929238 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.929127 5055 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9dvcx\"/\"kube-root-ca.crt\"" Oct 30 02:23:10.929238 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.929162 5055 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9dvcx\"/\"default-dockercfg-v8hcx\"" Oct 30 02:23:10.934618 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.934591 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x98l2\" (UniqueName: \"kubernetes.io/projected/042f82d2-417d-44b3-b691-8786eefad700-kube-api-access-x98l2\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:10.934774 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.934679 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-podres\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:10.934774 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.934714 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-lib-modules\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:10.934869 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.934819 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-proc\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:10.934921 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:10.934867 5055 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-sys\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.035654 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.035607 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x98l2\" (UniqueName: \"kubernetes.io/projected/042f82d2-417d-44b3-b691-8786eefad700-kube-api-access-x98l2\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.035852 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.035687 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-podres\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.035852 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.035710 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-lib-modules\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.035852 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.035747 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-proc\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.035852 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.035791 5055 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-sys\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.035852 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.035834 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-podres\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.036057 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.035843 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-lib-modules\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.036057 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.035887 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-sys\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.036057 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.035887 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/042f82d2-417d-44b3-b691-8786eefad700-proc\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.057185 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.057140 5055 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x98l2\" (UniqueName: \"kubernetes.io/projected/042f82d2-417d-44b3-b691-8786eefad700-kube-api-access-x98l2\") pod \"perf-node-gather-daemonset-67pds\" (UID: \"042f82d2-417d-44b3-b691-8786eefad700\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.135605 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.135498 5055 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.434106 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.434077 5055 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds"] Oct 30 02:23:11.434883 ip-10-0-134-254 kubenswrapper[5055]: W1030 02:23:11.434854 5055 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod042f82d2_417d_44b3_b691_8786eefad700.slice/crio-882d2485dd21c2874f295cfc48d083845c88ff11831a39a9c347a1a8b9bd34d1 WatchSource:0}: Error finding container 882d2485dd21c2874f295cfc48d083845c88ff11831a39a9c347a1a8b9bd34d1: Status 404 returned error can't find the container with id 882d2485dd21c2874f295cfc48d083845c88ff11831a39a9c347a1a8b9bd34d1 Oct 30 02:23:11.931497 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.931462 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" event={"ID":"042f82d2-417d-44b3-b691-8786eefad700","Type":"ContainerStarted","Data":"dc7bdba03d0ef6c12f269988a103aeee059879398ba2fe1a50f9f95fbe44f6db"} Oct 30 02:23:11.931932 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.931511 5055 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" event={"ID":"042f82d2-417d-44b3-b691-8786eefad700","Type":"ContainerStarted","Data":"882d2485dd21c2874f295cfc48d083845c88ff11831a39a9c347a1a8b9bd34d1"} Oct 30 02:23:11.931932 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.931654 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:11.973396 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:11.973346 5055 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" podStartSLOduration=1.973307541 podStartE2EDuration="1.973307541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:11.971695329 +0000 UTC m=+3959.292652381" watchObservedRunningTime="2026-10-30 02:23:11.973307541 +0000 UTC m=+3959.294264582" Oct 30 02:23:13.438577 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:13.438543 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jhxpw_58571a89-2cf1-44fa-8ef4-a052d02e1a82/manager/0.log" Oct 30 02:23:13.581017 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:13.580987 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jhxpw_58571a89-2cf1-44fa-8ef4-a052d02e1a82/manager/0.log" Oct 30 02:23:13.803283 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:13.803257 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-kzdph_0f1832b1-6bc7-4bf2-8dd9-de2bd8d55d41/networking-console-plugin/0.log" Oct 30 02:23:14.312164 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:14.312126 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cvjxp_36ce2879-a790-49fb-85d7-e6cd921a42c1/manager/0.log" Oct 30 02:23:14.465184 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:14.465138 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cvjxp_36ce2879-a790-49fb-85d7-e6cd921a42c1/manager/0.log" Oct 30 02:23:14.774151 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:14.774112 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78cc6d847c-4nbvj_b505b494-3911-42aa-80f7-71cfece2b6da/manager/0.log" Oct 30 02:23:14.818653 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:14.818620 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cbpcp_5dce57b6-a313-4051-a33b-7300d36d13ab/server/0.log" Oct 30 02:23:14.929954 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:14.929914 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78cc6d847c-4nbvj_b505b494-3911-42aa-80f7-71cfece2b6da/manager/0.log" Oct 30 02:23:14.969040 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:14.969007 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cbpcp_5dce57b6-a313-4051-a33b-7300d36d13ab/server/0.log" Oct 30 02:23:15.158034 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:15.157940 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/2.log" Oct 30 02:23:15.166164 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:15.166133 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-tw27b_b6cb0d17-b098-408e-92b6-b5e81a8bb9cf/console-operator/3.log" Oct 30 02:23:16.472841 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:16.472802 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-nnzpp_0855692d-c388-4913-a361-7af6c41bac7a/download-server/0.log" Oct 30 02:23:16.568154 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:16.568117 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-92htv_a93ddb49-90a6-416d-981a-afd68b09076b/manager/0.log" Oct 30 02:23:16.620518 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:16.620488 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tf79s_916568e2-0b52-47e0-a7bd-3635c0ceca97/proxy/0.log" Oct 30 02:23:16.701795 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:16.701764 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-92htv_a93ddb49-90a6-416d-981a-afd68b09076b/manager/0.log" Oct 30 02:23:16.742282 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:16.742183 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tf79s_916568e2-0b52-47e0-a7bd-3635c0ceca97/proxy/0.log" Oct 30 02:23:17.741940 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:17.741908 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-z5d5r_98bbc2c8-1ab4-4ebc-aea0-fffcaf91d4db/volume-data-source-validator/0.log" Oct 30 02:23:17.946783 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:17.946754 5055 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-67pds" Oct 30 02:23:18.322726 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:18.322696 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jhxpw_58571a89-2cf1-44fa-8ef4-a052d02e1a82/manager/0.log" Oct 30 02:23:18.410724 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:18.410692 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jhxpw_58571a89-2cf1-44fa-8ef4-a052d02e1a82/manager/0.log" Oct 30 02:23:19.176422 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:19.176382 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cvjxp_36ce2879-a790-49fb-85d7-e6cd921a42c1/manager/0.log" Oct 30 02:23:19.223265 ip-10-0-134-254 kubenswrapper[5055]: E1030 02:23:19.223220 5055 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-jz2gt" containerName="extractor" Oct 30 02:23:19.292840 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:19.292799 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cvjxp_36ce2879-a790-49fb-85d7-e6cd921a42c1/manager/0.log" Oct 30 02:23:19.601827 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:19.601784 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78cc6d847c-4nbvj_b505b494-3911-42aa-80f7-71cfece2b6da/manager/0.log" Oct 30 02:23:19.656751 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:19.656713 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cbpcp_5dce57b6-a313-4051-a33b-7300d36d13ab/server/0.log" Oct 30 02:23:19.704154 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:19.704112 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78cc6d847c-4nbvj_b505b494-3911-42aa-80f7-71cfece2b6da/manager/0.log" Oct 30 02:23:19.753764 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:19.753723 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cbpcp_5dce57b6-a313-4051-a33b-7300d36d13ab/server/0.log" Oct 30 02:23:20.948135 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:20.948110 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdpcp_1e24a38d-9f35-40bf-bb90-57fee63d8ac0/dns/0.log" Oct 30 02:23:21.066193 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:21.066163 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qdpcp_1e24a38d-9f35-40bf-bb90-57fee63d8ac0/kube-rbac-proxy/0.log" Oct 30 02:23:21.308569 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:21.308538 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9gckm_b47f8e4a-5060-47bf-8bec-722ccff0fa51/dns-node-resolver/0.log" Oct 30 02:23:21.399970 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:21.399927 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-92htv_a93ddb49-90a6-416d-981a-afd68b09076b/manager/0.log" Oct 30 02:23:21.488713 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:21.488675 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-92htv_a93ddb49-90a6-416d-981a-afd68b09076b/manager/0.log" Oct 30 02:23:21.523695 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:21.523660 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tf79s_916568e2-0b52-47e0-a7bd-3635c0ceca97/proxy/0.log" Oct 30 02:23:21.629810 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:21.629735 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tf79s_916568e2-0b52-47e0-a7bd-3635c0ceca97/proxy/0.log" Oct 30 02:23:22.882096 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:22.882049 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6d99f44fbc-h76gr_26443b6e-20d3-47e9-b0fd-e21d38b42e7c/registry/0.log" Oct 30 02:23:23.100953 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:23.100924 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-kckpv_c9801105-8f8a-45d0-b2c1-23468a212347/node-ca/0.log" Oct 30 02:23:23.341593 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:23.341563 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jhxpw_58571a89-2cf1-44fa-8ef4-a052d02e1a82/manager/0.log" Oct 30 02:23:23.387892 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:23.387860 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jhxpw_58571a89-2cf1-44fa-8ef4-a052d02e1a82/manager/0.log" Oct 30 02:23:24.245016 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:24.244942 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cvjxp_36ce2879-a790-49fb-85d7-e6cd921a42c1/manager/0.log" Oct 30 02:23:24.289191 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:24.289135 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-cvjxp_36ce2879-a790-49fb-85d7-e6cd921a42c1/manager/0.log" Oct 30 02:23:24.710223 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:24.710167 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78cc6d847c-4nbvj_b505b494-3911-42aa-80f7-71cfece2b6da/manager/0.log" Oct 30 02:23:24.755405 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:24.755367 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78cc6d847c-4nbvj_b505b494-3911-42aa-80f7-71cfece2b6da/manager/0.log" Oct 30 02:23:24.776573 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:24.776542 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cbpcp_5dce57b6-a313-4051-a33b-7300d36d13ab/server/0.log" Oct 30 02:23:24.805838 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:24.805802 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-cbpcp_5dce57b6-a313-4051-a33b-7300d36d13ab/server/0.log" Oct 30 02:23:25.696308 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:25.696277 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fvvsbg_551f197c-a32d-4dcb-b4f7-d8106d500e9b/istio-proxy/0.log" Oct 30 02:23:26.575368 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:26.575309 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-92htv_a93ddb49-90a6-416d-981a-afd68b09076b/manager/0.log" Oct 30 02:23:26.594424 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:26.594390 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-92htv_a93ddb49-90a6-416d-981a-afd68b09076b/manager/0.log" Oct 30 02:23:26.692128 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:26.692098 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tf79s_916568e2-0b52-47e0-a7bd-3635c0ceca97/proxy/0.log" Oct 30 02:23:26.720251 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:26.720222 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-tf79s_916568e2-0b52-47e0-a7bd-3635c0ceca97/proxy/0.log" Oct 30 02:23:27.426176 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:27.426131 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fvvsbg_551f197c-a32d-4dcb-b4f7-d8106d500e9b/istio-proxy/0.log" Oct 30 02:23:28.445196 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:28.445160 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fvvsbg_551f197c-a32d-4dcb-b4f7-d8106d500e9b/istio-proxy/0.log" Oct 30 02:23:28.583403 ip-10-0-134-254 kubenswrapper[5055]: I1030 02:23:28.583374 5055 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6455b4886fvvsbg_551f197c-a32d-4dcb-b4f7-d8106d500e9b/istio-proxy/0.log"